Web scraping: Reliably and efficiently pull data from pages that don't expect it
Why Stack Overflow recommends it
Stack Overflow users link it when someone asks how to scrape or log in to websites in Python and needs a broad, tool-comparing overview.
Recommended by 4 people across 5 different months, 2010 to 2014.
What it is
A 2h39m PyCon tutorial by Asheesh Laroia on extracting data from web pages and HTML forms in Python. It covers parsing pages, choosing among tools such as urllib2, mechanize and lxml, and techniques like parallel downloads that make scrapers faster and more stable.
What people said
Recommends Diego Castro · 2010-03 · Stack Overflow · on “Which is best in Python: urllib2, PycURL or mechanize?”
I think this talk (at pycon 2009), has the answers for what you're looking for (Asheesh Laroia has lots of experience on the matter). And he points out the good and the bad from most of your listing Scrape the Web: Strategies for programming websites that don't expect it (Part 1 of 3) [LINK] Scrape the Web: Strategies for programming websites that don't expect it (Part 2 of 3) [LINK] Scrape the Web: Strategies for programming websites that don't expect it (Part 3 of 3) [LINK] From the PYCON 2009 schedule: Do you find yourself faced with websites that have data you need to extract? Would your life be simpler if you could programmatically input data into web applications, even those tuned to r
Recommends root · 2012-10 · Stack Overflow · on “Using Beautiful Soup to find Descriptions”
watch this [this video] , also what does it mean "ran into trouble"? Do you get an error message, what kind of error is it? Adding this information will save people's time and improves your chance of getting an answer.
Recommends tomasz74 · 2012-11 · Stack Overflow · on “Web scraping - web login issue”
In your question you mention CasperJS but you tag question as python. If you want to use python as a lenguage you can check video [this video] describing different tools for web scraping For menaging webpages with login you can use mechanize [LINK] Sample code from above webside: br = mechanize.Browser() # Explicitly configure proxies (Browser will attempt to set good defaults). # Note the userinfo ("joe:password@") and port number (":3128") are optional. br.set_proxies({"http": "joe:password@myproxy.example.com:3128", "ftp": "proxy.example.com", }) # Add HTTP Basic/Digest auth username and password for HTTP proxy access. # (equivalent to using "joe:password@..." form above) br.add_proxy_password
Recommends root · 2013-04 · Stack Overflow · on “How to get content from HTML page and print in Python?”
Using lxml.html [LINK] : In [1]: import lxml.html as lh In [2]: doc = lh.parse(' [LINK] ') In [3]: doc.xpath('.//div[@class="day main"]//em[@class="max"]/text()') Out[3]: ['21'] But , I guess what you really want to do is take a look at a web scraping tutorial [this video] .
Recommends Ape-inago · 2014-08 · Stack Overflow · on “Best way for a beginner to learn screen scraping by Python”
Not a direct answer to your question, but you might want to check out this video: [this video] it covers a lot of the more advanced topics about screenscraping. It comes at it from the perspective of python, but for the most part it does it in terms of theory and is largely language agnostic.