Agenty scripts are JavaScript functions, written for a special run-time environment that automate the modification of agent output result or the input data entered by user, either by manually or selecting a URL list or source agent. Scripts are often interpreted (rather than compiled). There are two types of scripts available to add with each agent:

Pre-Processing
Post-Processing

Pre-Processing

Script to modify input on the fly - This script executes before the agent starts the job, and should be used to manipulate the agent input -

/**
  *   Modify the agent input data before the scraping job is started
  *   For example, to generate Dates (today + next 7 day date), 
  *   for a flight search website scraping
  * */

module.exports = function modifyInput(input) {
     input.forEach(row => {
             row.url += `?checkIn=${new Date())}`;         
     })
     return input;
 }

The Pre-Processing script helps to modify the agent input automatically. For example, if we want to extract the data from an airline website which requires the Check In as current date and Check Out as plus 7 days in the current date in the query-string. We can write a Pre-processing script to modify the static input URL http://example.com/flight-search into dynamically generated url with current date and +7 days automatically on manual jobs or on scheduled jobs and can write a script to generate URL in format like:- http://example.com/flight-search?check-in=01\05\2018&check-out=01\12\2018

Post-Processing

Script to modify output on the fly - This script executes after the agent job has been completed and before integrations, this script should be used to manipulate the output table : -

 /**
  *   Transform the scraped output after the scraping job is completed
  *   For example, to add conditional field, replace values, 
  *   currency conversion etc
  * */
 
 module.exports = function modifyOutput(data) {
     data.forEach(row => {
         if (row.currency == 'USD') {
             row.price = Number(row.price) * 70;
         }
     })
     return data;
 }

The Post-Processing script helps to modify the agent’s output automatically. For example, if we have an price column in my agent, the result is USD currency. I can use the post-processing script to convert the price into INR automatically.

Add a logical field

In this example, we will use the post-processing script to add a logical field in an agent output result, we can simply add a column in the result array and then set the value of that field.

For example, I have an agent with 3 fields (ProductName, Price, Tax), now I want to add a 4th field TotalPrice where Price + Tax should be the value of TotalPrice:

 module.exports = function modifyOutput(data) {
     data.forEach(row => {
             row.TotalPrice= Number(row.Price) + Number(row.Tax);         
     })
     return data;
 }

Filter data

In this example we will filter the rows where price is greater than 100. So just use the Filter function on array of objects passed to modifyOutput function and apply the filter.

 module.exports = function modifyOutput(data) {
     const result = data.filter(row => Number(row.price) > 100);
     return result;
 }

Concatenate

The Concatenate function allows you to combine text from different cells into one cell. In this example, I will show you how to combine multiple values to make it into single cell using template strings -

 module.exports = function modifyOutput(data) {
     data.forEach(row => {
         row.Heading = `The rank of book ${row.Title} is ${row.Rank}`
     })
     return data;
 }

Auto fill

Auto fill the blank cells down taking the values from previous non-empty cell. In this example, I will show you how to to fill the empty cells by copying the value from last non-empty cell similar to excel autofill to drag the cell values.

 module.exports = function modifyOutput(data) {
     let lastValue = "";
     data.forEach(row => {
         if (row.name == '') {
            row.name = lastValue;
         }else{
            lastValue = row.name;
         }
     });
     return data;
 }

Scripting

Pre-Processing

Post-Processing

Add a logical field

Filter data

Concatenate

Auto fill

Before auto fill

After auto fill

Signup now to get 100 pages credit free