How to combine 2 or more scraping agents result in one agent?

To combine multiple web scraping agent results with exact same fields, we can use the Agenty post-processing script feature to merge multiple data tables and then set the agent result back to Agenty.

// Get the current job output as DataTable
var table = Agenty.Cloud.GetAgentResult("first agent id");

// Get running job id
var jobId = table.ExtendedProperties["job_id"];

// Get other agents data
var table2 = Agenty.Cloud.GetAgentResult("second agent id");

// merge both tables

// set job id
table.ExtendedProperties["job_id"] = jobId;

// set the result back to table

Remember to run this agent in the end, as it will download the 1st agent result to combine. So, the first agent must be completed before the post-processing script is executed.

For example, I have 2 agents to scrape electronics and hardware categories from an eCommerce website and want to combine the final result in a single CSV.

I can use this script to meet my needs.

  • Go to the agent page
  • Click on the Scripting tab
  • Switch on the post-processing script
  • Enter the script with and change the agent id GetAgentResult("here") in get result function
  • Save and run

Signup now to get 100 pages credit free

14 days free trial, no credit card required!