How to scrape hidden data from collapsible element?

Collapsible elements, commonly implemented using HTML, CSS, and JavaScript, provide a way to organize and present content in a compact manner. In this article I will show you how to use a custom JavaScript function to expend these elements for scraping the data from collapsed HTML

The hidden content within collapsible elements poses a challenge to common web scrapers because traditional web scraping techniques often rely on parsing the raw HTML of a page.

So, hidden content is typically not loaded in the initial HTML source especially in websites built with new front-end technologies like Angular, Vue and React meaning that a basic scraping attempt might miss crucial data from collapsible elements.

Here are the step-by-step instructions to learn how to click on expand button programmatically to scrape data from hidden HTML -

Inspect the element structure

Use your web browser’s developer tools to inspect the structure of the collapsible element. Identify the HTML elements responsible for triggering the expansion and containing the hidden content to write your JavaScript function.

• Press F12 on go to setting > More tools > Developer tools
• Go to sources tab
• Create a new snippet to test the JavaScript function

Inspect the Collapsible elements917×592 29 KB

Trigger the element expansion

Simulate a user action by programmatically triggering the element’s expansion using JavaScript. This can usually be achieved by dispatching a click event to the trigger element.

For example, here is my JavaScript function to click on all primary buttons using the querySelectorAll to find the button, and clicking on all using the click() method.

var elements = document.querySelectorAll('.btn-primary');
for(const el of elements){
    el.click();
}

Custom script implementation

Once the element’s expansion JavaScript function is ready, open your agent and paste the script in the Wait for > script editor to run it automatically after each page render to manipulate the DOM before data scraping.

Custom script implementation for element's expansion1167×762 60.5 KB