How can I create an API to scrape website data?

I want to access my scraping agent via API, so that I can send the request from my android app with parameters to scrape a website.

How can I extract the data using dynamically submitted URLs? and then start the scraping to extract the final result in JSON.

There are code sample available for Microsoft .Net to mange the web scraping API (See here), and also available on Github you can replicate in your program.

Here is sample in curl :

Send a HTTP PUT request to add urls in your scraping agent

curl -X PUT -H "Content-type: application/json" -d '{
    "type": "MANUAL",
    "data": [
        "http://website.com/products",
        "http://website.com/products/page-1.html",
        "http://website.com/products/page-2.html"
    ],
    "reference": null
    }' 'https://api.agenty.com/v1/inputs/{AGENT ID}?apikey={API KEY}'

Then send a POST request to start the agent

curl -X POST -H "Content-type: application/json" -d '{
    "agent_id":"{AGENT ID}"
    }' 'https://api.agenty.com/v1/jobs/scraping/async?apikey={API KEY}'

And GET request finally to fetch the scraped result in JSON

curl -X GET -H "Content-type: application/json" 
   'https://api.agenty.com/v1/jobs/{JOB_ID}/result?apikey={API KEY}&offset=0&limit=1000'
14 days free trial

Automate your business with advanced, fully-featured agents on Agenty. Fast, scalable and no-code web automation tool for scraping, change monitoring and more...

Sign up for free →

No credit card required