Python Programming Tutorial - 26 - How to Build a Web Crawler (2/3)
Why Stack Overflow recommends it
Stack Overflow users cite it as the tutorial they were following when their crawler code broke, usually because the demo site changed or the selectors no longer match.
Recommended by 1 people across 1 different months, 2015 to 2021.
What it is
Part 2 of a three-part beginner tutorial by Bucky Roberts (thenewboston) on building a simple web crawler in Python. It walks through using requests and BeautifulSoup to loop over pages, pull out links by class, and print their hrefs and titles.
What people said
Recommends PythonPadawan · 2019-08 · Stack Overflow · on “How to access and parse the source frame in a python web scraping application on a classic asp website”
I'm trying to create a web-crawler application that downloads a bunch of assorted PDFs from SAI using this tutorial as my basis [this video] I've tried get(url) on the website I believe I want to get the source from Authentication code (not included) .... def subscription_spider(max_pages): page = 1 while page <= max_pages: url= ' [LINK] ' + str(page) source_code = session.get(url) # need to get the frame source!! # extra code to find the href for the frame than get the frame source # [LINK] plain_text = source_code.text #source_code.text printed is not the one we want playFile = open('source_code.text', 'wb') for chunk in r.iter_content(100000): playFile.write(chunk) playFile.close() soup = Beau
Cites yoOwhatUP · 2015-03 · Stack Overflow · on “How do I build a Web Crawler using Python 3?”
I have been watching 3 videos of thenewboston on how to make a webcrawler. It seems they are outdated and the links are not there. I would be thankfull if someone could fullfill the first part of the tutorial. This is as far as I have come. I have tried different websites but to no avail. import requests from bs4 import BeautifulSoup def my_spider(max_pages): page = 1 while page <= max_pages: url = """Here I tried different websites""" + str(page) source_code = requests.get(url) plain_text = source_code.text soup = BeautifulSoup(plain_text) for link in soup.findALL("a", {"class" : "item-name"}): href = """example site like following: [LINK] """ + link.get("href") print(href) page += 1 print(
Cites Dante · 2016-06 · Stack Overflow · on “Stuck Coding a Python web crawler using BeautifulSoup”
regarding this video by bucky roberts on programming a webcrawler in python: [this video] Here is my question: If i want to crawl a particular item but it isnt inside of < a> < /a> how can i do it? for ex. I inspected the site and found this is the code of the the info i want (i want the href and the title just like in the bucky's video): < td headers="categorylist_header_title" class="list-title" > < a href="BLABLABLABLA.HTML" > blablabliblableblu < /a> < /td> Following the instruction i notice that the element "class="list-title" inside of is the one i need to crawl, but when i use for link in soup.findAll('a',{'class': 'list-title'}): it does not work, i think because it isnt inside of < a>, h
Cites Vincent · 2017-10 · Stack Overflow · on “Unable to crawl Reddit's NBA page”
I'm new to web crawling and want to learn how to use beautifulsoup to integrate it on a mini project. I was following thenewboston tutorial on beautifulsoup on his [this video] then got stuck trying to crawl off of Reddit. I want to crawl titles and links on each of the NBA news on Reddit/r/nba [LINK] but didn't have any success. Only thing that return in the terminal was "Process finished with exit code 0". I have a feeling it was to do with my selections? Any guidance and help would be greatly appreciated. This is the original code, didn't work: import requests from bs4 import BeautifulSoup def spider(max_pages): page = 1 while page <= max_pages: url = ' [LINK] ' + str(page) source_code = reque
Cites Hirusha Adikari · 2021-05 · Stack Overflow · on “why doesn't my web scraper work? Python3 - requests, BeautifulSoup”
I have been following this python tutorial [this video] for a while, and I made a web scrawler, similar to the one in the video. Language: Python import requests from bs4 import BeautifulSoup def spider(max_pages): page = 1 while page <= max_pages: url = ' [LINK] ' + str(page) source_code = requests.get(url) plain_text = source_code.text soup = BeautifulSoup(plain_text, 'html.parser') for link in soup.findAll('a', {'class':'item-title'}): href = link.get('href') title = link.string print(href) page += 1 spider(1) And this is the output that the program gives: PS D:\development> & C:/Users/hirusha/AppData/Local/Programs/Python/Python38/python.exe "d:/development/Python/TheNewBoston/Python/one/web