Bulk Web Scraper & Scraping API

Paste a list of URLs, define the fields you want with CSS selectors, and scrape every page with JavaScript rendering and optional proxies.

Bulk web scraper

Paste one URL per line. Requests run one by one in your browser. 5 of 5 free requests left. Sign in for more.

Fields to scrape

All matches

Render JavaScript

Wait until

Residential proxy

API reference

Add the fields you want above (a name and a CSS selector each), paste your URLs, and the scraper visits every page and returns the matching values as JSON. Pick Text, HTML, or an Attribute such as href or src, and turn on All matches to collect lists like every product on a category page.

Turn off JavaScript rendering for static sites to get results faster, or turn on a residential proxy to scrape geo-restricted pages. For recurring jobs, pagination, and scheduling, build a scraping agent. Every option is documented in the Scrape API reference.

Features

CSS selectors
Any CSS selector, including attribute and pseudo-class selectors.
Text, HTML, or attribute
Extract text, inner HTML, or any attribute value.
Lists of items
Collect every match, or just the first one.
JS rendering toggle
Render SPAs in a browser, or skip it for speed.
Residential proxies
Geo-targeted proxies for blocked or regional sites.
Absolute URLs
href and src values resolved to full URLs.
Bulk scraping
Paste a list of URLs and scrape them in one run.
JSON export
Download all results as a single JSON file.

Use cases

E-commerce price, stock, and review monitoring
Real-estate and rental listing aggregation
Job board aggregation and salary benchmarking
Travel rate and availability comparison
Collect titles, meta descriptions, and headings for SEO audits

API examples

View full API reference

Scrape a list of products with CSS selectorsbash

curl -X POST https://api.agenty.ai/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://scrapingsandbox.com",
    "render": false,
    "fields": [
      { "name": "name", "selector": ".product-card .product-name", "multiple": true },
      { "name": "price", "selector": ".product-card .price", "multiple": true },
      { "name": "vendor", "selector": ".product-card .vendor", "multiple": true },
      { "name": "link", "selector": "a.product-card", "extract": "ATTR", "attribute": "href", "multiple": true }
    ]
  }'

Scrape through a residential proxy in the USbash

curl -X POST https://api.agenty.ai/v1/scrape \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/pricing",
    "proxy": "us",
    "selectors": { "plan": "h2", "price": ".price" }
  }'

Scrape many URLs in Node.jsjavascript

const urls = [
  'https://scrapingsandbox.com/product/1',
  'https://scrapingsandbox.com/product/2',
  'https://scrapingsandbox.com/product/3',
];

for (const url of urls) {
  const res = await fetch('https://api.agenty.ai/v1/scrape', {
    method: 'POST',
    headers: {
      'Authorization': 'Bearer YOUR_API_KEY',
      'Content-Type': 'application/json',
    },
    body: JSON.stringify({
      url,
      fields: [
        { name: 'name', selector: 'h1' },
        { name: 'description', selector: 'meta[name="description"]', extract: 'ATTR', attribute: 'content' },
      ],
    }),
  });
  const { result } = await res.json();
  console.log(url, result.name);
}

Scrape a page in Pythonpython

import requests

res = requests.post(
    "https://api.agenty.ai/v1/scrape",
    headers={"Authorization": "Bearer YOUR_API_KEY"},
    json={
        "url": "https://example.com",
        "selectors": {"heading": "h1", "description": "p"},
    },
)
print(res.json()["result"])

How Agenty compares

Feature	Agenty	ScrapingBee	ScraperAPI	ZenRows
CSS selector support	Yes	Yes	Yes	Yes
Online tool, no code	Yes	No	No	No
JavaScript rendering	Yes	Yes	Yes	Yes
Residential proxies	Yes	Yes	Yes	Yes
Free tier	Yes	Yes	Yes	Yes

Frequently asked questions

What is the Web Scraping API?

The Agenty Scrape API extracts data from any web page using CSS selectors. You send a URL and a list of fields, and get back a JSON object with the value of each field. See the API reference for every option.

How do I scrape multiple URLs at once?

Paste your URLs into the scraper at the top of this page, one per line, define your fields, and click Scrape pages. Each URL is scraped with the same fields and you can download all results as JSON. To collect URLs first, use the link extractor or sitemap crawler.

When should I turn off JavaScript rendering?

Set render: false when the data is already in the HTML the server sends, which is true for most blogs, news sites, and server-rendered stores. Requests are faster. Keep rendering on for single-page apps built with React, Vue, or Angular.

How do I use a proxy?

Pass proxy: true to route the request through a residential proxy, or a two-letter country code such as "us" or "de" to pick the location. You can also turn on Residential proxy in the online tool and pick a country.

I do not know CSS selectors. What can I use instead?

Use AI data extraction: describe what you want in plain English and the AI finds it on the page.

Is there a free tier?

You can scrape 5 pages without an account, and all accounts start with free credits. Visit our pricing page for current limits.