Agenty Firestore integration allows you to send your web scraping agent’s result to Google Cloud Firebase Firestore NoSQL database by connecting to your project id, collection using service credentials.
Firestore is a real-time NoSQL database built on Google cloud infrastructure to scale automatically.
The Agenty Firestore integration allows you to import your agent’s result to selected Firestore project database collection automatically.
Create Service Account
Agenty requires the service credential JSON file to connect with your Google cloud services. So, the first step to start using Firestore plugin in Agenty is to get your service credential file.
If you have the key file already with Firestore permission? You can skip this step and jump to configure the plugin directly.
A service account represents a google cloud service identity, such as code running on Compute Engine VMs, Server-less etc.
- Login to your Google cloud account and go to IAM & Admin page
- Click on the Service Accounts in left menu to go to service account page
- Click on the Create Service Account button to create a new service account. Then enter the name and description of your choice.
- Add a role to give full read and write access to Firestore Database
5. Click on the continue button and find the Create key button on next page. Click on the Create key button to download a JSON file that contains the private key and other information for server-to-server connection.
6. Click on the Create button and it will download the key file in JSON format.
Configure Firestore Plugin
- Login to your Agenty account
- Go to Plugins page
- Click on the Add button against the Firestore plugin
- Firestore plugin configuration page will open
- Select the agent where you want to attach this plugin to trigger on job completion.
- Then upload the credentials file and enter the project id, collection name in plugin configuration
- The primary key field is optional. This can be used to specify a primary key field from your agent to
overwriteexisting documents (if any). If not specified, Firestore will auto-generate primary key to insert as new document - Save this plugin configuration to attach it with the selected agent
When a primary key field is specified, Agenty will use that field value and send an overwrite request to Firestore. If the document does not exist, it will be created. If the document does exist, its contents will be overwritten with the newly provided data
Execution
Go to the agent page where you’ve attached this Firestore plugin. For example, I attached it to a web scraping agent with name : Firestore plugin test
And, start that agent. (You may also start the web scraping task using our API)
It will take a few seconds to complete the job, depending on the number of URLs you are crawling in this agent. So, wait for the completion.
Logs and Preview
Click on the logs tab and scroll to the last lines to see the plugin execution logs. You will see the message with Added or Updated document with id successfully.
| DATE | LEVEL | MESSAGE |
|---|---|---|
| 2019-10-27 09:56:01.4120 | TRACE | Firestore plugin started with timeout: 15 minutes |
| 2019-10-27 09:56:09.9411 | TRACE | Added document with ID : dQw9rBUZky5imz8CQabQ successfully |
| 2019-10-27 09:56:09.9411 | TRACE | Added document with ID : wNIhVclyetms8SeIOEie successfully |
| 2019-10-27 09:56:09.9411 | TRACE | Added document with ID : 2fWdILu7Mx0tPBI3gfoI successfully |
| 2019-10-27 09:56:09.9411 | TRACE | Added document with ID : UmZ3ImbYCL2qeDAoSYlm successfully |
| 2019-10-27 09:56:09.9411 | TRACE | Added document with ID : q1ouQlLle5MXQmUKR221 successfully |
| 2019-10-27 09:56:09.9411 | TRACE | Added document with ID : 98TephxtXjd3uMIm4Zi8 successfully |
Go to your Firestore project page - https://console.firebase.google.com/ and select your project. Then go to database > collection to see the result sent by Agenty.
