Questions tagged [beautifulsoup]
Beautiful Soup is a Python package for parsing HTML/XML. The latest version of this package is version 4, imported as bs4.
32,480
questions
0
votes
0
answers
25
views
Elements missing in saved html from playwright
Im trying to scrape the prices from buff.163 but i cant find them in the html that was saved. My code looks like this:
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
...
0
votes
2
answers
58
views
extract value from span
I want to extract a snow depth value from a weather site to a dataframe. (https://www.yr.no/nb/sn%C3%B8dybder/NO-46/Norge/Vestland) Specifically the snow depth for the Jordalen - Nåsen area.
Screen ...
0
votes
1
answer
19
views
How to remove a tag from an element of BeautifulSoup?
I have a page like this:
...
<div class="myclass">
<p>
text 1 to keep<span>text 1 to remove</span>and keep this too.
</p>
<p>
text 2 to keep&...
1
vote
1
answer
22
views
how to get anchors href value and nested page anchors href value in using concurrent.futures?
I am trying to fetch the anchor tag href value and nested urls with concurrent.futures?
I want to reducing the time to fetch the urls using below code.
it's taking time to fetch the URLs using below ...
1
vote
1
answer
30
views
Identify text with span elements using Python BeautifulSoup
I have html text that I am trying to clean up using "soup". However, I need to not only identify which text segments are contained by certain span elements of the class='highlight', but also ...
-3
votes
2
answers
44
views
TypeError: write() argument must be str, not Tag [duplicate]
Here is my code:
html = open("meetup.html", "r")
from bs4 import BeautifulSoup
soup = BeautifulSoup(html, 'html.parser')
file = open('myfile.txt', 'w')
for elem in soup.find_all(...
2
votes
1
answer
42
views
How to extract text from HTML with BeautifulSoup?
from bs4 import BeautifulSoup
import requests
website = 'https://www.klavkarr.com/data-trouble-code-obd2.php?dtc=p0000-p0299#dtc'
result = requests.get(website)
content = result.text
soup = ...
0
votes
1
answer
27
views
Beautifulsoup: find the location of a string in the soup
I am trying to scrap data from a web page generated by a search engine. For this purpose, I study the data structure of the document using a sample query and the output result. The output contains a ...
1
vote
1
answer
33
views
how to modify a BeautifulSoup object
Let's presume I have the following piece of code
text = '<link href="link">'
souped = BeautifulSoup(text, 'html.parser')
tags = souped.find_all()
link_modified_tags = ResultSet()
...
2
votes
1
answer
29
views
Webscraping Entry id from the Uniprot database search result
Dear Stack Overflow community,
I'm working on a project where I need to extract entry IDs from the UniProt website's search results. Specifically, I'm looking to retrieve IDs like 'O00622' from HTML ...
0
votes
0
answers
18
views
Enable cookies and javascript with BS4
I have coded a scraper using BS4 to get some stats off a betting website. The scraper was working but now I am getting this error saying I need to enable JS and cookies to continue:
<!DOCTYPE html&...
1
vote
1
answer
49
views
Trouble getting an html table from an aspx website
I am trying to extract the table from this website:
https://serviciosede.mineco.gob.es/indeco/reports/verSerieGraf.aspx/?codigo=230400&frec=-1
The table starts with column headers "Fecha"...
-3
votes
0
answers
16
views
how to get discounted price using meta property tag in beautifulsoup python? [closed]
My code is:
price = soup.find("meta", attrs={"property": "product:price:amount"}).get("content")
I want to get discounted price from a webpage like below:
...
-1
votes
0
answers
42
views
While converting scraped data into csv file hotel price and other things showing in special word [closed]
import pandas as pd
import requests
from bs4 import BeautifulSoup
Hotel = []
Description = []
Prices = []
for page_number in range(1, 16):
url = f"https://www.airbnb.co.in/s/Jaipur--India/...
1
vote
2
answers
32
views
How to scrape section of HTML and extract text?
I am trying to extract the string 2 and a half stars, 4688 Ratings in python from the below HTML element.
I used this command and got this set of data. I can't figure out how to extract this portion ...
-3
votes
0
answers
48
views
Is there any way to use BeautifulSoup to webscrape from Bible.com? [closed]
I'm trying to webscrape a verse from a specific version of the Bible in Turkish from Bible.com, the Kutsal Kitap Yeni Çeviri 2001, 2008. I'm able to use Beautiful Soup to webscrape from most websites, ...
0
votes
2
answers
44
views
Basic webscraping-but I cannot solve this problem
I want to practice webscrapping and follow some video. That video said that:
import requests
from bs4 import BeautifulSoup
url = "https://comic.naver.com/webtoon/list?titleId=811721&tab=wed&...
0
votes
1
answer
45
views
Web Scraping returns T&C link
This URL has some CSV data files I'd like to use for analysis under the folder 'usage-stats'. However, I'm unable to scrape data from this page. When I try using the below code, I can only see the ...
-1
votes
0
answers
28
views
Web Scraping - Cant find data on a html page
I am trying to scrape the following web page
'https://www.racingpost.com/profile/horse/3763274/kerkyra-island/form'
In particualar I would like to scrape the data in the red box in the photo attached ...
-2
votes
0
answers
42
views
CSS properties supported by Microsoft Word [closed]
I'm converting a .html file to .docx format, and I want to know if there are some properties like <thead> with display: table-header-group to repeat the header of a table in each page.
I want to ...
-2
votes
1
answer
58
views
How to find div with specific class with bs4
I'm trying to scrape a specific class of divs from an HTML document using bs4. However, when I use the find_all() method, I do not get back the divs that I want despite the fact that I can see those ...
-1
votes
1
answer
32
views
Trouble Scraping a Page Full of JavaScript Code Using Python and Selenium [duplicate]
I am currently working on a web scraping project using Python and Selenium, and I'm encountering difficulties with a particular webpage that seems to be heavily reliant on JavaScript. When I inspect ...
-1
votes
1
answer
106
views
+50
Unable to load JavaScript and got pyppeteer error from webpage with requests
I'm trying to scrape, after loggin, this webpage: https://saywow.me
If I use only BeautifulSoup and requests I get Please enable JavaScript to continue using this application. error page
So, I decided ...
-1
votes
0
answers
14
views
Getting the contents of a specific table using BeautifulSoup
I am trying to get the content in the table on this website (data.ers.usda.gov/reports.aspx?ID=17828); however, I am failing to locate the table so I can proceed. I have tried numerous code but none ...
0
votes
0
answers
20
views
Beautiful Soup returning XX instead of actual values [duplicate]
I am trying to Webscrape a website using requests and beautiful soup.
But I am getting 'XXX' instead of the text within the tags.
code:
url = 'https://www.discover.com/student-loans/'
req = opener....
-1
votes
1
answer
45
views
Web Scraping, load more data into a HTML
I am looking to scrape data from the following web page https://www.racingpost.com/bloodstock/sales/catalogues/5/2023-12-04
I am using beatiful soup and have requested the html as seen below. The soup ...
-1
votes
0
answers
19
views
Can't upload media to wordpress using Python script and wordpress_xmlrpc
Pretty new to this so please keep it simple!
Basically I have data I have scraped with a Python script from one URL which includes a title, description, video and images.
I have another script which ...
-2
votes
0
answers
32
views
How do i make this load more images? (Python) [closed]
I have this function I've written to search for a specific word on google images, and then acquire all the urls for the images it detects (except the firts one, which is always the google logo):
def ...
-2
votes
0
answers
30
views
Need help to download images by searching for images on that website [closed]
So I am trying to webscrape images from the PIER digital library of images https://peir.path.uab.edu/library/search.php. (It is a public library). But I seem to encounter a problem. So I have a list ...
1
vote
2
answers
46
views
Unable to fetch list from angular website using beautiful soup
I am trying to parse nptel website (url) to fetch all courses listed on it.
Each course correspond to app-courses-card tag. The DOM looks like this:
I am unable to get hold of these tags with ...
0
votes
2
answers
65
views
How to find main price and discounted price in a webpage using selenium and python?
I am trying to find a way to find main price and also discounted price in a webpage but I can get just one of them and I need a good pattern or method to extract all price and discounted prices from ...
0
votes
1
answer
48
views
Python Beatifulsoup only scrapes the 1st page of Amazon reviews, someone knows how to extract all ideally without a proxy?
I am using Beautifulsoup on Python to try to scrape all Amazon reviews for a product, but it only extracts the 1st page (9 reviews). This code seem to have worked in the past for other users, maybe I ...
1
vote
1
answer
57
views
how to extract table from a webpage using its id?
I was trying to gather some data for a data analysis project concerning italian football. Unfortunately I can't extract one specific table from the web page, even though the id is correct.
The code is ...
-2
votes
0
answers
19
views
Parsing (and updating) paragraph with embedded tags [closed]
I'm trying to parse and modify the words in a paragraph (that may have other tags (i.e. formatting)) with a XHTML file.
For example,
XHTML=b"""<?xml version="1.0" encoding=&...
0
votes
1
answer
82
views
Error using BeautifulSoup code to retrieve info from cooking website. Issues parsing HTML
I am attempting to create a program that will bring back the Title, URL, Ingredients, and Recipe Instructions from recipes listed in the allrecipes.com sitemap. However, I'm missing something for the ...
-3
votes
0
answers
29
views
Problems with parsing hash [closed]
There is a website https://mrush.mobi/welcome
I want to create a bot for the site, but there is a problem with the hash attribute
Tell me how to get the correct hash?
image
I tried to write something ...
0
votes
1
answer
42
views
How Can I Extract a Value from HTML?
I am building automation to deal with email alerts that we receive. The final step is to be able to extract the username involved in the alert and from research it looked like it should be fairly easy ...
-2
votes
1
answer
31
views
How to get the all anchor tags current page and nested pages url? [closed]
I want to get the attribute value href in a anchor tags current page and nested from the URL
https://www.tradeindia.com/ pages which is matches the value .*([0-9]+).html in href attribute value.
Lile ...
1
vote
2
answers
45
views
Getting allways the same values when iterating using Selenium?
i try to collect some data using the following code:
import time
import os
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from ...
-2
votes
0
answers
39
views
'NoneType' object has no attribute 'contents' [closed]
problem: using .replace(",", "") method with .contents method of Beautiful-Soup to convert 103,121 to integer 103121 return
using function alone doesnt give error but when be used ...
0
votes
2
answers
32
views
List of all static-ralated tags and elements, and how to get all static from a beautifulsoup
I working on a beautifulsoup finding of all elements that are related to static.
What is the list of all static-related tags and elements?
Is there any other way apart from finding by multiple ...
1
vote
3
answers
54
views
How to extract only the first items from several span tags using Beautiful Soup?
I only began to learn Beautiful Soup recently. I'm stuck trying to extract only the first items from a part of the html code I recently scraped from a site. I simplified the original html code as ...
-1
votes
0
answers
31
views
My scraper search app is returning 400 Bad Request and I am unsure why
I'm trying to create a search engine which returns keyword results from findaphd.com. This is my Python code and I am unsure what I'm doing wrong. I'm quite new to coding so apologies if this is a ...
0
votes
3
answers
105
views
How to parse xml to pandas dataframe using beatiful soup
I have:
xml_data = """<address_details sequence_number="1" match_indicator="L">
<address_input id="ADI" address_details="street address ...
-2
votes
0
answers
29
views
The content of scraped div is empty (Dynamic JS script) [closed]
Could someone please help me scraping information from transfermarkt?
This is an example url: https://www.transfermarkt.com/georginio-wijnaldum/transfers/spieler/49499
This is how it looks in DevTools:...
-1
votes
0
answers
69
views
Traceback (most recent call last), TypeError: 'NoneType' object is not callable
Project was to scrap the product information from Flipkart like Product Name, Price, Rating, Specifications.
All extracted data stored in data frame by using pandas and then export to CSV format.
When ...
-1
votes
1
answer
35
views
Is it possible to use selenium to input to a website, then beautiful soup to parse the HTML info, then output to a desktop tool? [closed]
I am working on a weather tool and I need to be able to input a city name to a website, once that website give me the informaiton for the citys historical weather information I need to be able to pull ...
-1
votes
1
answer
41
views
How do I change a particular part of a XML which has been read into Python and then create a new XML with this change? [duplicate]
So I am using BeautifulSoup to read and parse an XML below:
from bs4 import BeautifulSoup
with open(r"....TestXML.xml", 'r') as f:
data = f.read()
bs_data = BeautifulSoup(data, "...
-4
votes
1
answer
83
views
How to web scrape a number with python
I am trying to web scrape a simple number from this website
I tried this code but get an empty variable:
import requests
from bs4 import BeautifulSoup
url = "https://www.quantalys.com/Recherche?...
-1
votes
1
answer
41
views
Trip Advisor Rating count returns 'None' - Python Webscraping
I am tyring to scrape the Trip-Adivors number of reviews in Yellowpages.
Here is the complete code from start to bottom.
import requests
from bs4 import BeautifulSoup
import pandas as pd
import time
...