Table of contents

How to Scrape Data Behind a Login with Playwright

On this page

Playwright can be particularly useful when scraping data accessible only behind a login wall. This article shows a practical example of such a case.

Scraping expenses on Amazon

For our example, we will be logging in to our Amazon account and scraping the price off each order in the previous year, then adding them all up to show us our total Amazon expenditures over that period of time.

A combination of UI automation and scraping will allow us to first log in to the platform, and then to retrieve the information about all our orders.

Playwright


const { chromium } = require('playwright')
;(async () => {
  const currency = process.env.CURRENCY
  const amazonUrl = process.env.AMAZON_URL
  const amazonUser = process.env.AMAZON_USER
  const amazonPassword = process.env.AMAZON_PASSWORD

  const browser = await chromium.launch()
  const page = await browser.newPage()

  await page.goto(amazonUrl)

  await page.setViewportSize({ width: 1200, height: 1822 })

  await page.click(
    '#nav-signin-tooltip > .nav-action-button > .nav-action-inner'
  )

  await page.type('#ap_email', amazonUser)
  await page.click('#continue')
  await page.type('#ap_password', amazonPassword)

  await page.click('#signInSubmit')

  await page.click('#nav-link-accountList > .nav-long-width')

  await page.click(
    '.ya-card__whole-card-link > .a-box > .a-box-inner > .a-row > .a-column > div'
  )

  await page.click('#a-autoid-1-announce > .a-dropdown-prompt')

  await page.click('#orderFilter_2')

  let filteredPrices = []
  let prices = []

  await page.waitForSelector('.a-normal')
  const pages = await page.$$('.a-normal')

  for (const singlePage of pages) {
    await singlePage.waitForSelector('.a-last')
    prices = await singlePage.$$eval(
      '.a-column:nth-child(2) .a-color-secondary.value',
      (nodes) => nodes.map((n) => n.innerText)
    )

    filteredPrices = filteredPrices.concat(
      prices.filter((price) => price.includes(currency))
    )
    await singlePage.waitForSelector('li.a-last')
    await singlePage.click('li.a-last')
  }

  const cleanPrices = filteredPrices.map((x) => {
    return x.replace(',', '.').replace(/[^\d.,-]/g, '')
  })

  const totalExpense = cleanPrices.reduce(
    (a, b) => parseFloat(a) + parseFloat(b),
    0
  )
  console.log(`Total expense for the year: ${currency} ${totalExpense}`)

  await browser.close()
})()

⚠️ This example is only intended for learning purposes. Always make sure the website you are planning to scrape allows such behaviour.

Run the above examples as follows, making sure to choose the right Amazon URL and currency:

macOS

CURRENCY=EUR AMAZON_URL=https://www.amazon.de AMAZON_USER=<YOUR_AMAZON_USERNAME> AMAZON_PASSWORD=<YOUR_AMAZON_PASSWORD> node scraping-example-purchases.js

Windows

SET CURRENCY=EUR
SET AMAZON_URL=https://www.amazon.de
SET AMAZON_USER=<YOUR_AMAZON_USERNAME>
SET AMAZON_PASSWORD=<YOUR_AMAZON_PASSWORD>
node scraping-example-purchases.js

Under the hood, Amazon can change quite quickly. You might need to adjust the locators and/or flow slightly to have the script work for you.

⚠️ Websites might restrict headless browser traffic in order to protect their users from fraud. 2FA will also interfere with the script if enabled.

Takeaways

We can scrape information available behind a login wall with Playwright.
Some websites might not allow scraping. Always make sure you check their terms of service beforehand.

How to Scrape Data Behind a Login with Playwright

Scraping expenses on Amazon

Takeaways

Further reading