Web Scraping with Node.js & Puppeteer (🌋 rants included, no extra charge)
Why Stack Overflow recommends it
Stack Overflow users link it as a starting walkthrough for Puppeteer scraping and cite its $$eval and map technique for pulling many elements into an array.
Recommended by 1 people across 1 different months, 2018 to 2021.
What it is
A 41-minute live-coding session from the DevTips channel in which the hosts build a Node.js web scraper with Puppeteer (headless Chrome) to collect business data for a database. It is loosely structured, with fika breaks and rants on Google, Prettier and VS Code along the way.
What people said
Recommends Peyter · 2020-01 · Stack Overflow · on “Scraping carousell with puppeteer”
As @hardkoded stated, document is not something that is out of the box in puppeteer, it's dogma in the browser, but not in Node.js. You also do not need to for each in Node.js. The Map Technique outlined in this video [this video] is very helpful and quick. I'd make sure also to keep await on your loop or map technique, because the function is asynchronous so you want to make sure the promise comes back resolved. Map technique An extremely fast way to get many elements into an array from a page is to use a function like below. So instead of getting an array of the elements and then looping them for their properties. You can create a function like this below using $$eval and map. The result is a f
Cites Norfeldt · 2018-10 · Stack Overflow · on “JS Puppeteer wait for page load to complete”
After seeing this [this video] I got inspired to play a bit around with it. But I seem to have made the wrong choice of a website as a starter project. const puppeteer = require('puppeteer') ;(async () => { const browser = await puppeteer.launch() const page = await browser.newPage() await page.goto(' [LINK] ') page.once('load', () => { const drugs = page .evaluate(() => [...document.querySelectorAll('div.entity-link')].map(item => item) ) .catch(err => console.log(err)) console.log(drugs[0]) }) await browser.close() })() I have googled around and lost track of the different things I have tried.. My perception of my problem is that I don't call the evaluate at the right time - when the page is lo
Cites Logchop · 2021-02 · Stack Overflow · on “web scraping with puppeteer does not find the CSS tag”
im starting to learn web scraping in javascript with puppeteer. I found a video that I liked that showcases puppeteer and I'm trying to scrape the same information as the video( link [this video] ). the page has changed a little from the video so I used what I think are the correct tags. the problem comes when I try to find the "h3" tag. the tag exists in the DOM but my code refuses to acknowledge its existence but works "fine" when looking for the "h2" tag. what I want to know is why my code does not retrieve it. web page: [LINK] // normal things to launch it const puppeteer = require("puppeteer"); (async() => { const browser = await puppeteer.launch(); const page = await browser.newPage(); cons