Scraping JavaScript pages with Scrapy and Splash
Why Stack Overflow recommends it
Stack Overflow users link it when answering or asking about Scrapy spiders that return empty results because content is built by JavaScript, as a quick walkthrough for setting up scrapy-splash.
Recommended by 3 people across 3 different months, 2017 to 2021.
What it is
A short tutorial from Zyte (formerly Scrapinghub) in its "Learn Scrapy" series showing how to use the Splash rendering service and the scrapy-splash plugin so Scrapy spiders can scrape JavaScript-generated pages. The description lists the exact settings needed for the scrapy-splash middlewares and dupe filter.
What people said
Recommends M3RS · 2017-08 · Stack Overflow · on “Scraping node with specific text using scrapy and xpath”
Thanks! Now I understand! This video was also helpful: Scraping JavaScript pages with Scrapy and Splash [this video]
Recommends Ayushman Koul · 2018-03 · Stack Overflow · on “Extract all href using css/xpath selectors”
You can use reference of Scrapy Tutorial [this video] to write code for this page since it involves javascript to generate the content of body.
Recommends Ridgebeck · 2018-10 · Stack Overflow · on “Scrapy - Extracting specific data via CSS query doesn't work”
Thank you all for your quick responses. It was indeed a problem with dynamically generated content via javascript. Scrapy's build in "browser" only returns raw html code without executing the javascript. Using splash and the plugin scrapy-splash worked! Here is a good and short video that helped me to add scrapy-splash: [this video]
Cites J. Does · 2017-12 · Stack Overflow · on “Log in with form loaded with Ajax (scrapy): selenium vs scrapy-splash”
To scrape the webpage I want, I need to log in. To access the log in form, I have to click on a button. This button makes an AJAX request, which displays the form. I use Scrapy, with a middleware to pass my request through Tor and privoxy. I have seen I could use selenium to click on the button but then (if I got it right), I can parse the response obtain by selenium with scrapy. I have seen, I could use scrapy-splash [LINK] with seems a better option. I have followed several tutos including this one [this video] , but I can't come up with a way to use scrapy-splash to click on a button to load the log in form and then to submit this form. My settings.py: (I didn't set HTTP_PROXY) DOWNLOADER_MIDD
Cites goblin_rocket · 2018-05 · Stack Overflow · on “Scrapy & Splash not returning anything from javascript page”
Im currently following the official docs as well as a Youtube video to scrape javascript pages with Scrapy and their splash js rendering service. [LINK] [this video] I have Docker installed on my Mac and run it as per the official doc instructions: docker run -p 8050:8050 -p 5023:5023 scrapinghub/splash I then have this demo code taken from the Youtube video: import scrapy from scrapy_splash import SplashRequest class Demo_js_pider(scrapy.Spider): name = 'jsdemo' def start_request(self): yield SplashRequest( url = ' [LINK] ', callback = self.parse, ) def parse(self, response): for quote in response.css("div.quote"): yield { 'text': quote.css("span.text::text").extract.first(), 'author': quote.css
Cites Elle518 · 2021-05 · Stack Overflow · on “Scrape javascript generated result with scrapy-splash”
I'm trying to scrape the result generated by a javascript function when you introduce some acronym in a dictionary search box. This is the code I'm using: import scrapy from scrapy_splash import SplashFormRequest class SedomSpider(scrapy.Spider): name = 'sedom-spider' url_s = ' [LINK] ' formdata = {'sigla': 'AA'} def parse(self, response): return SplashFormRequest( url=self.url_s, formdata=self.formdata, callback=self.after_login ) def after_login(self, response): print(response.body) I obtain the html body, but the generated result ( <span id="resultado">...</span> ) I should see is not in it... I've also tried: import scrapy from scrapy_splash import SplashRequest, SplashFormRequest class
Cites Mert · 2021-06 · Stack Overflow · on “Scraping images in a dynamic, JavaScript webpage using Scrapy and Splash”
I am trying to scrape the link of a hi-res image from this link [LINK] but the high-res version of the image can only be inspected upon clicking on the mid-sized link on the page, i.e after clicking "Click here to enlarge the image" (on the page, it's in Turkish). Then I can inspect it with Chrome's "Developer Tools" and get the xpath/css selector. Everything is fine up to this point. However, you know that in a JS page, you just can't type response.xpath("//blah/blah/@src") and get some data. I install Splash (with Docker pull) and configure my Scrapy setting.py files etc. to make it work (this [this video] helped. no need to visit the link unless you wanna learn how to do it) . ...and it worked