To combine multiple web scraping agents result with exact same fields, we can use the Agenty post-processing script feature to merge multiple data tables and then set the agent result back to Agenty.
// Get the current job output as DataTable var table = Agenty.Cloud.GetAgentResult("first agent id"); // Get running job id var jobId = table.ExtendedProperties["job_id"]; // Get other agents data var table2 = Agenty.Cloud.GetAgentResult("second agent id"); // merge both tables table.Merge(table2); // set job id table.ExtendedProperties["job_id"] = jobId; // set the result back to table table.SetAgentResult();
Remember to run this agent in last, as it will download the 1st agent result to combine. So, the first agent must be completed before the post-processing script is executed.
For example, I have 2 agents to scrape electronics and hardware category from an eCommerce website and want to combine the final result in a single CSV.
I can use this script to meet my need.
- Go to the agent page
- Click on the Scripting tab
- Switch on the post-processing script
- Enter the script with and change the agent id
GetAgentResult("here")in get result function
- Save and run