How to get/set cookies, sessionStorage and localStorage in Puppeteer?

In this post, I will show you how to use Agenty’s scraping agent to get/set cookies, sessionStorage and localStorage using Puppeteer and Chromium headless browsers for advanced web scraping.

Cookies, Session storage and Local storage are used for storing data in the browser. Websites needed to keep some data locally for session management, login, location and personalization. To see what kind of data is stored by any website, you need to open developer tools and go to the Storage tab to find out.

session storage on headless browser870×476 17.9 KB

Cookies

Cookies is text or encrypted content which is stored in your browser’s temporary memory and allows websites to remember certain information that you need. Some websites may not work properly if cookies are disabled in your browser.

You can use the page.setCookie() method in puppeteer to set cookies for any website. Remember, you need to first navigate to a page via Puppeteer before you can set cookies on a particular website. So, always use the page.goto() first and then set cookies.

Here is example code to set cookies -

// Go to the page set cookies
// Get the status code, title, cookies in result

module.exports = async ({ page, request }) => {
    const response = await page.goto(request.url);            
    
    console.log(response.status());
    const cookies = [{
      'name': 'cookie_1',
      'value': 'value 1'
    },{
      'name': 'cookie_2',
      'value': 'value 2'
    }];

    await page.setCookie(...cookies);
    
    const result = {
        url : response.url(),
        cookies: await page.cookies(request.url),
        status : response.status(),
        title : await page.title(),
    };
    
    return {
        data : result,
        type : 'application/json'
    };   
};

Puppeteer cookie storage1239×697 39.7 KB

Session Storage

Session storage is an in-memory browser storage that stores data for a short time. The data can only be accessed during the current session in the same tab only.

Here is example code to get/set sessionStorage using Puppeteer -

Set

To set the some data in session storage

await page.evaluate(() => {
        let sessionData = {key : value};
        window.sessionStorage.setItem('data', JSON.stringify(sessionData));
    })

Get

To get the data from session storage by key

await page.evaluate(() => {
       const data = window.sessionStorage.getItem('data');
       console.log(JSON.parse(data));
    })

Local Storage

Local storage is a mechanism that allows websites to store data locally in the browser. Using localStorage, a website can store key-value pairs where the value can be a complex object or simple string accessible in all browser tabs.

Here is example code to get/set localStorage using Puppeteer -

Set

To set the some data in local storage

await page.evaluate(() => {
        let sessionData = {key : value};
        window.localStorage.setItem('data', JSON.stringify(sessionData));
    })

Get

To get the local storage data by name

await page.evaluate(() => {
        window.localStorage.getItem('data'); 
        console.log(JSON.parse(data));
    })