I want to scrape all the options under the select element to scrape product variants available on the eCommerce website which I am scraping. Web scraping drop-down lists become a little easy with Agenty.
Let’s take an example…
Example HTML
<select name="sizes">
<option value="S">Small</option>
<option value="M">Medium</option>
<option value="L">Large</option>
<option value="XL">Extra-Large</option>
</select>
I tried to use the “.sizes” selector and it’s not scraping anything. And I don’t want full text, instead the option value of the select item.
Here “sizes” is not a class, so you cannot use it with dot (.) to match this element. You should be using the attribute selector select[name="sizes"] option
and value
as your attribute to scrape all values.
If you want to know more about scraping selected options from the dropdown menu, you can go through this how to extract selected options from dropdown list!
Here is Github example to try out - Forum #16