Firestore Integration

Agenty Firebase workflow allows you to send your web scraping agent’s result to Google Cloud Firebase Firestore NoSQL database by connecting to your project id, collection using service credentials.

Firebase is a real-time NoSQL database built on Google cloud infrastructure to scale automatically.

The Agenty Firebase workflow allows you to import your agent’s result to selected Firestore project database collection automatically.

Create Service Account

Agenty requires the service credential JSON file to connect with your Google cloud services. So, the first step to start using Firestore plugin in Agenty is to get your service credential file.

If you have the key file already with Firestore permission? You can skip this step and jump to configure the workflow directly.

A service account represents a google cloud service identity, such as code running on Compute Engine VMs, Server-less etc.

  1. Login to your Google cloud account and go to IAM & Admin page
  2. Click on the Service Accounts in left menu to go to service account page
  3. Click on the Create Service Account button to create a new service account. Then enter the name and description of your choice.
  4. Add a role to give full read and write access to Firestore Database


5. Click on the continue button and find the Create key button on next page. Click on the Create key button to download a JSON file that contains the private key and other information for server-to-server connection.

6. Click on the Create button and it will download the key file in JSON format.

Now it’s time to see how to configure Firebase workflow in agenty

Configure Firebase Workflow

  1. Login to your Agenty account
  2. Go to the Workflows tab

  1. Click on the +Create a workflow button

  1. Firebase plugin configuration page will be open
  2. Select the agent where you want to attach this workflow to trigger on job completion.

  1. Then upload the credentials file and enter the project id, collection name in workflow configuration

  1. The primary key field is optional. This can be used to specify a primary key field from your agent to overwrite existing documents (if any). If not specified, Firestore will auto-generate primary key to insert as new document
  2. Save this workflow configuration to attach it with the selected agent

When a primary key field is specified, Agenty will use that field value and send an overwrite request to Firestore. If the document does not exist, it will be created. If the document does exist, its contents will be overwritten with the newly provided data

Execution

Go to the agent page where you’ve attached this Firebase workflow. For example, I attached it to a web scraping agent with name : Firestore plugin test

And, start that agent. (You may also start the web scraping task using our API)

Start web scraping agent with FireStore workflow

It will take a few seconds to complete the job, depending on the number of URLs you are crawling in this agent. So, wait for the completion.

Logs and Preview

Click on the logs tab and scroll to the last lines to see the plugin execution logs. You will see the message with Added or Updated document with id successfully.

DATE LEVEL MESSAGE
2019-10-27 09:56:01.4120 TRACE Firestore plugin started with timeout: 15 minutes
2019-10-27 09:56:09.9411 TRACE Added document with ID : dQw9rBUZky5imz8CQabQ successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : wNIhVclyetms8SeIOEie successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : 2fWdILu7Mx0tPBI3gfoI successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : UmZ3ImbYCL2qeDAoSYlm successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : q1ouQlLle5MXQmUKR221 successfully
2019-10-27 09:56:09.9411 TRACE Added document with ID : 98TephxtXjd3uMIm4Zi8 successfully

Go to your Firestore project page - https://console.firebase.google.com/ and select your project. Then go to database > collection to see the result sent by Agenty.

Signup now to get 100 pages credit free

14 days free trial, no credit card required!