Can multiple Diffbot Extract APIs be used in a single crawl?
No. Only one Extract API may be used at a time with a single crawl.
If you wish to process multiple types of pages through separate APIs, your options are:
Use the Analyze API
The Analyze API will automatically determine the page-type of each page crawled, and structure the data from supported pages. This content can be filtered using DQL API, or the JSON downloaded in full and filtered using the type field.
Set up Multiple Crawls
To explicitly use multiple APIs for crawling a single site, you'll need to set-up multiple crawls, each using an independent API. You can use Crawl's multiple mechanisms for controlling/narrowing your crawls to ensure that each separate crawl job only processes the right type of pages.
Updated over 2 years ago