I have few websites needs to scrape regularly for price monitoring. How can I estimate how long it will take to run the scraper and extract prices from all those websites?
The best way to estimate how long it will take to complete your web scraping job is getting answer to following questions -
- How many pages you are crawling?
- What is your pricing plan?
- What is the website URL you are crawling? How fast is that?
- Are you using Agenty proxies or your own?
And then run a small scraping jobs with let’s say 1000 URLs to get an idea to see how that performs .
If I look for general web scraping benchmark of Agenty we did a while ago - Then, here is a simple test result we performed on 1000 pages in different plans with 1, 3, and 5 concurrent workers respectively.