How to scrape dynamically submitted URLs with the Agenty API

To scrape URLs submitted by your Android app or another program, use the Agenty API to update your scraping agent, start a scraping job, and fetch the results in JSON. The workflow uses three requests: PUT to submit URLs, POST to start scraping, and GET to retrieve the results.

Microsoft .NET code samples for managing the web scraping API are available in the documentation (See here) and on Github. You can adapt those samples for your program, or use the curl examples below.

1. Submit URLs to your scraping agent

Send an HTTP PUT request to add the URLs you want to scrape. Replace the sample URLs in data with the URLs submitted by your app. Replace {AGENT ID} and {API KEY} with your agent ID and API key in each request.

curl -X PUT -H "Content-type: application/json" -d '{
    "type": "manual",
    "data": [
        "http://website.com/products",
        "http://website.com/products/page-1.html",
        "http://website.com/products/page-2.html"
    ],
    "reference": null
    }' 'https://api.agenty.com/v1/inputs/{AGENT ID}?apikey={API KEY}'

2. Start the scraping job

Send a POST request with your agent ID to start the agent asynchronously.

curl -X POST -H "Content-type: application/json" -d '{
    "agent_id":"{AGENT ID}"
    }' 'https://api.agenty.com/v1/jobs/scraping/async?apikey={API KEY}'

3. Fetch the scraped results in JSON

Send a GET request for the scraping job’s results, replacing {JOB_ID} with the job ID. This example requests up to 1,000 results, starting at offset 0.

curl -X GET -H "Content-type: application/json" 
   'https://api.agenty.com/v1/jobs/{JOB_ID}/result?apikey={API KEY}&offset=0&limit=1000'