• Crawlbot: resolved an issue where IP-address-only webhooks would not receive notifications.
• Crawlbot: improved link spidering/harvesting resilience to markup errors and other invalid HTML source.
• Fixed an issue where custom APIs would not display in Crawlbot and Bulk Processing dashboard.

• Improved link-detection when returning page links via our &fields=links argument.
• Improved support for and handling of the srcset (and sizes) image attributes in all APIs.
• Added detection of Afrikaans (af) in the humanLanguage field.
• Improved duplicate detection in the Diffbot Global Index.

• The beta category field has been added to the Product API. See documentation.
• All extraction APIs now support the sending of completely custom headers using X-Forward- terminology. Previously only four defined headers were supported.

• In the Article and Discussion APIs' tags element, DBPedia uri values are now properly URL-encoded.
• Fixed an issue when sorting by date in the Search API.
• Various improvements and fixes to the Global Index

• The Account API now tracks Global Index search calls/requests.
• Improved SKU detection and extraction in the Product API.
• Article API: Added support for the start attribute (ol elements) and data- attributes in normalized HTML.
• In the Article API, identified image captions will no longer be returned in the text field content.
• Various improvements to replacement rule regular expressions in Custom APIs.
• PDF processing improvements.

• Product API: overriding the sku, mpn or related fields using custom rules will now affect the productId field as well.
• Crawls using the Analyze API will now correctly index video pages.
• Improved the reliability of the fields=links argument in all Automatic APIs.

Updates to our rendering engine to properly support more Unicode scripts

• Updates to our status page for improved coverage and reliability.
• Crawlbot crawls can now have repeat settings adjusted or added after a crawl completes.
• Fixed a Crawlbot issue wherein users could completely erase the seeds field.

• POSTing to our APIs is speedier, particularly when content includes slow-loading third-party assets.
• Crawlbot now has limited support for crawling/processing content across multiple domains.

• Improvements to background image detection and extraction across all Automatic APIs. This resolves many issues with proper extraction from sites that use background CSS properties for image delivery.
• Improvements to specification extraction in the Product API.
• Improvements to HTML <figure> parsing in the Article API.