Scraping product, prices and stock information from an eCommerce competitor’s websites is one of the real business uses of web scraping tool. And most of the eCommerce companies rely on web scraping software for price comparison, and use this technique to compare their price to find out if they are offering the best price in the competition or not. And then strategic planning to increase margin-profit.

This entire process is called competitive pricing analysis or pricing intelligence in business to monitor competitor prices and plan business strategy to adjust margins. In this tutorial we will learn how to extract price from a simple product list website (category or search result page) from an eCommerce website to learn the basics of price extraction from competitor websites using Agenty web scraping tool.

Setup

To extract any site with Agenty’s data scraping tool, we need to create a scraping agent for that website first. So in order to setup a scraping agent, I’ve already installed the chrome extension. Now:

Go to the competitor website page you are looking to extract products from and then enable the extension by clicking on the extension icon on the right corner of the browser.
Click on the New button to add fields and “Asterisk (*)” button generates selectors for any HTML element you are looking to extract.

Let’s see how to use price scraping tool to collect competitor’s price and other details. Here, I am going to show how to scrape these 4 fields mentioned below to extract from this example product listing page:

ProductName
ProductPrice
ProductImage
ProductCartLink

Field 1 - ProductName

The CSS selector worked for product_name is .list-group-item-heading which is displaying all 6 product names correctly in preview :

[
 {
   "ProductName": "Product A"
 },
 {
   "ProductName": "Product B"
 },
 {
   "ProductName": "Product C"
 },
 {
   "ProductName": "Product D"
 }
]

Field 2 - ProductPrice

To extract the price, I found that .lead is the correct CSS selector to extract the prices of all 6 products available in this page along with their USD currency symbol. See the preview in the screenshot below :

Field 3 - ProductImage

The CSS selector worked for ProductImage is .list-group-image and the attribute as src which is displaying the 6 images for all 6 products on this page correctly, as we can see in the extracted result preview :

Field 4 - ProductCartLink

To scrape the ProductCartLink .btn-success is a good css selector and the attribute as href which is displaying links for all 6 products correctly in extracted result preview :

Result Preview

Once you are done with setting up all the fields, click on the Preview button which is on the top of the Agenty’s extension to see the result preview :

[
  {
    "ProductName": "Product A",
    "ProductPrice": "$21.40",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/118"
  },
  {
    "ProductName": "Product B",
    "ProductPrice": "$16.80",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/119"
  },
  {
    "ProductName": "Product C",
    "ProductPrice": "$30.00",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/120"
  },
  {
    "ProductName": "Product D",
    "ProductPrice": "$16.25",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/121"
  },
  {
    "ProductName": "Product E",
    "ProductPrice": "$19.00",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/122"
  },
  {
    "ProductName": "Product F",
    "ProductPrice": "$23.50",
    "ProductImage": "http://placehold.it/400x250/000/fff",
    "ProductCartLink": "/cart/123"
  }
]

Save & Execution

When you are satisfied with the extracted result preview click on the “Save” button to save the web scraper agent to Agenty cloud application where you can use the features like scheduling, batch url crawling and more advanced options to automate the screen scraping to extract prices from websites.

Note : You need to enter your Agenty Email and Password to save the web scraping agent (https://www.agenty.com)

Now login to Agenty and you’ll see your agent in your “Scraping Agents” list or on your “Dashboard” where you can start the agent to scrape the data or schedule that to run when you want. In the screenshot below I ran the agent and it extracted the product and pricing details . As shown in the table below.

Want to try on your own?

Sign in to Agenty and click on New Agent button
Here you would find this same agent in “Sample Agents” list to get from

Check it out and let me know what you think in comments below, or request a demo if you are an Enterprise and looking for managed services to monitor your competitors.

How to extract prices from competitor websites?