Web crawling software for the entire website

Advanced crawling settings options to put you in control of your web crawling experience like never before. With customizable options including depth, maximum pages, and the ability to follow sitemaps, you can tailor your web crawling agent to match your specific needs and objectives.

• Follow sitemaps automatically
• Respect the robots.txt
• Add multiple fields
• Follow internal or external pages
• Maximum time, links and result option to auto-stop after certain conditions

Web crawling rules for targeted data extraction from the web. With the ability to specify complete URLs, parts of URLs, REGEX patterns, and even exclude specific pages, you have unparalleled control over the webpages you crawl.

• Multiple rules options to match any to skip or crawl a page
• Negative rules to prefix with exclamation mark (!) to skip crawling
• REGEX pattern to follow matching pattern only
• Priority rules to crawl important pages first
• Processing rules to extract data from specific pages

Crawl an entire website automatically by discovering internal links, XML sitemaps and pagination to build a complete site map of every reachable URL, with depth and canonical tracking.

• Automatic internal link discovery
• XML sitemap and sitemap index parsing
• Pagination and infinite scroll handling
• Duplicate and canonical URL detection
• Broken link and status code reporting

Crawl modern JavaScript websites and single page applications with a real headless Chrome browser so dynamic content, lazy-loaded sections and AJAX data are fully rendered before extraction.

• Headless Chrome rendering engine
• Wait for selector or fixed delay
• Execute custom JavaScript on page
• Click, scroll and type actions
• Crawl behind login with session cookies

Distributed cloud crawlers with concurrency control, automatic retries and IP rotation let you crawl millions of pages reliably without getting blocked or managing any servers.

• Parallel crawling with concurrency control
• Automatic retry on errors and timeouts
• Rotating residential and datacenter proxies
• Polite crawl delay and rate limiting
• Resume and incremental re-crawls

Download crawled data as CSV, TSV or JSON, or stream results straight into your stack with the REST API, webhooks and 12+ built-in integrations.

• Export as CSV, TSV or JSON
• REST API to launch crawls and fetch results
• Webhook notifications on completion
• Amazon S3, SFTP and Dropbox delivery
• Google Sheets and Zapier integration