Python Tutorial: Web Scraping with BeautifulSoup and Requests
Why Stack Overflow recommends it
Stack Overflow users link it as the tutorial they followed when learning BeautifulSoup, and askers often cite it when their copied scraping code breaks.
Recommended by 4 people across 3 different months, 2018 to 2021.
What it is
A 45-minute Python tutorial by Corey Schafer on scraping websites with the BeautifulSoup and Requests libraries. It walks through parsing HTML, pulling out headlines, summaries and embedded video links from a sample site, and saving the results to CSV.
What people said
Recommends Akono · 2019-11 · Stack Overflow · on “How to fix "TypeError: Every item in the descriptions list must be an AccessDescription"”
I am a novice python user trying to get a website's data to be parsed via Beautiful Soup. Whenever I run a get request, however, I get this certain error. I downloaded the html file and used it to write the processing, but I can't get the online updating working. I was going according to this youtube guide: [this video] import requests from bs4 import BeautifulSoup source=requests.get(' [LINK] ').text soup=BeautifulSoup(source,'lxml') print(r.text) runfile('C:/Users/Akono/Documents/RD2L Season 18/RD2L DB/scrape and csv.py', wdir='C:/Users/Akono/Documents/RD2L Season 18/RD2L DB') Traceback (most recent call last): File "<ipython-input-225-654df9a749a6>", line 1, in <module> runfile('C:/Users/Akono
Recommends Jay Lynch · 2020-04 · Stack Overflow · on “Multiple variable assignement in try block Python3”
I am going to refer you to Corey Schafer's video on web scraping and he deals with this issue. He can explain allot better than I could [this video]
Recommends Binx · 2020-04 · Stack Overflow · on “Web Scraping with Beautiful soup multiple duplicate tags”
This is my first time web scraping and am following this tutorial [this video] . I am using this website [LINK] to scrape information from. I am trying to grab the text that reads "89426 Green Mountain Road, Astoria, OR 97103. Phone: 503-325-9720." I noticed that I have multiple ul and li tags within my div class_=alert tag. So I am not sure how to grab a specific one. Here is what I have tried, but continue to get a different text from another group of ul / li . from bs4 import BeautifulSoup import requests source = requests.get(' [LINK] ').text soup = BeautifulSoup(source, 'lxml') noble_ridge = soup.find('div', class_='alert') information = noble_ridge.ul.li.text print(information) # print(soup
Recommends Matiiss · 2021-04 · Stack Overflow · on “How to Login to my Chegg account with python and extract the answer”
I already linked You to a tutorial [LINK] , it is pretty straight forward and I suppose You have to use `.post` function but also You have to provide Your own code that deals with this matter and what has failed. also I suggest then using BeautifulSoup [this video] , but start with requests [LINK]
Cites Pedro S · 2018-09 · Stack Overflow · on “Basic Python web scraping with BeautifulSoup”
I'm pretty new to coding and recently I've started looking into web scraping. I've been following this tutorial [this video] and also reading the BS4 documentation but I just can't see why my code isn't working. I'm trying to extract this post's headline [LINK] with the webscraper but it looks like it can't find any tags that match "('div', class_='header')" My code: import requests from bs4 import BeautifulSoup SOURCE = requests.get(' [LINK] ').text SOUP = BeautifulSoup('SOURCE', 'lxml') HEADER = SOUP.find('div', class_='header') HEADLINE = HEADER.h2.a.href print(HEADLINE) Error message: Traceback (most recent call last): File "WSCoreySchafer.py", line 10, in <module> HEADLINE = ARTICLE.h2.a.hre
Cites Alex Duciag · 2020-06 · Stack Overflow · on “How can I present IMDB info in html using Django and BeautifulSoup?”
I am kinda new to Django and WebScraping but I am having issues with presenting the info in html via Django. The function in Django does the following: first it asks the user to specify a movie genre and then it scrapes the IMDB address ( [LINK] ) with the added genre and returns, in this case, the top five examples. I want to use a function so I can present the info in HTML/DTL like so (this example is just pure Python and the way it prints the info is the way I want it to be represented in HTML): import requests from bs4 import BeautifulSoup source = requests.get(" [LINK] ").text soup = BeautifulSoup(source, "lxml") for article in soup.find_all("div", class_="lister-item-content")[:5]: hea
Cites Kevin Rodriguez · 2020-08 · Stack Overflow · on “TypeError: find() takes no argument inputs”
novice programmer in Python here! I'm trying to run the code below using python 3.8.5, but I keep getting the error present in title. Disclaimer: The code is not my original code, but it is from a YouTube video that was published in 2017. I will include its link here for reference: [this video] import requests import urllib import getpass source = requests.get(' [LINK] ').text soup = BeautifulSoup(source, 'lxml') article = soup.find('article').prettify() # print(article) # summary = article.find('div', class_='entry-content').p.text # print(summary) vid_src = article.find('iframe', class_= 'youtube-player')['src'] print(vid_src) I've been playing around with this for a few days, changing my pytho
Cites Wolf · 2021-01 · Stack Overflow · on “TypeError in HTML scraper Python. AttributeError: 'NoneType' object has no attribute 'get_text'”
Error checking You should really check for None results, see beautiful documentation of find() [LINK] If find() can’t find anything, it returns None None [LINK] itself represents the absence of a value. It therefore has also no attributes. Try None.get_text() in the python console to get the exact same message as in your question. of directly accessing elements in the hoped-for HTML output, take a look on the big picture ! Use print(soup) , the output is even easier to read if you redirect it into an HTML file. Firefox will render it like this: Amazon on bots [LINK] We see that in above case, it seems not too hard for amazon to detect that the access is not done by a human (perhaps caused by
Cites Marvel · 2021-07 · Stack Overflow · on “Get specific line with beautifulsoup using python”
I've watched this video which he scraped an article from his website, [this video] But the thing missing in the video is that he didn't explain how I can scrape a specific line in loop from the article. from bs4 import BeautifulSoup import requests import csv source = requests.get(' [LINK] ').text soup = BeautifulSoup(source, 'lxml') for article in soup.find_all('article'): headline = article.h2.a.text print(headline) summary = article.find('div', class_='entry-content').p.text print(summary) try: vid_src = article.find('iframe', class_='youtube-player')['src'] vid_id = vid_src.split('/')[4] vid_id = vid_id.split('?')[0] yt_link = f' [LINK] ' except Exception as e: yt_link = None print(yt_link) p
Cites Marvel · 2021-07 · Stack Overflow · on “Get specific line with beautifulsoup using python”
@MendelG When he loop the article he scrapped all the articles like the contents and src vids ..etc [this video] the thing that I want is I want to scrap a specific line from all these lines, hope you get it