The ultimate online collection toolbox: Combining RSelenium and Rvest - Part 1
Why Stack Overflow recommends it
Stack Overflow users link it as a practical walkthrough for getting RSelenium working, including the Docker and Selenium container setup that often trips people up.
Recommended by 2 people across 2 different months, 2020 to 2022.
What it is
Hanjo Odendaal presents a 90-minute R Consortium talk on web scraping in R, starting with the ethics and best practices of scraping and the robotstxt package. He then walks through rvest with exercises and introduces RSelenium for automating browsers locally and remotely.
What people said
Recommends seansteele · 2020-06 · Stack Overflow · on “Is RSelenium::remDr$open() actually suppose to open a browser window?”
Finally got it working and thought I would share what worked as there are a lot of this same question with no replies. So to answer my question, yes, remDr$open() should open a browser, but in a vnc container that you already have up and running. Here's my process to get there. First, make sure everything is up-to-date. So many problems could have been solved by making sure my programs, packages and library's where the newest. With that said, I read somewhere that a program (can't remember which) was having issues with Big Sur... Second, this resource was great about explaining how to setup Docker+Selenium on windows/linux machines and how to scrape using RSelenium. In it he warns that there
Recommends Rômulo Barros · 2020-12 · Stack Overflow · on “How to set up rselenium for R?”
Download Docker at [LINK] Run docker pull selenium/standalone-chrome-debug in terminal (or cmd for windows) In Docker Desktop's Dashboard, go to the "images" tab on the left. After that, you should see something like this: [LINK] Click Run A popup will appear. There, click on "Optional Settings" [LINK] Type 4445 on Ports. Click on the "plus" sign, type 5901 on the other input that will be created on Ports. It should look like the image below. After that, click Run. [LINK] Now, if you click on the Containers / Apps tab on the left, there should be something like this: [LINK] In Rs console, go: install.packages("RSelenium") library(RSelenium) remDr <- remoteDriver( remoteServerAdd = "localhost
Cites stevec · 2022-06 · Stack Overflow · on “selenium.common.exceptions.WebDriverException: Message: invalid session id using Selenium with ChromeDriver and Chrome through Python”
I got this error message because I was running Selenium in docker and I hadn't mounted enough swap memory, so it would crash after just a few pages. To fix this, I used the same docker command, but added -v /dev/shm:/dev/shm after docker run. If you had this docker run -d -p 5901:5900 -p 127.0.0.1:4445:4444 selenium/standalone-chrome then change to this docker run -v /dev/shm:/dev/shm -d -p 5901:5900 -p 127.0.0.1:4445:4444 selenium/standalone-chrome I found this info here [LINK] , and here [this video] . This will give the docker container more swap memory. From docker run --help it says: -v, --volume list Bind mount a volume --volume-driver string Optional volume driver for the container --volum