Questions tagged [beautifulsoup]
Beautiful Soup is a Python package for parsing HTML/XML. The latest version of this package is version 4, imported as bs4.
29,222
questions
0
votes
0
answers
6
views
How to save asynchronous tasks from parsing data to a dictionary of dictionaries?
I need to store and write to csv the parsed data in the form of the following structure, for example:{level1:'tc', 'level2':'ISO/IEC JTC 1 Information technology':{'level3''ISO/IEC JTC 1/SC Software ...
2
votes
0
answers
15
views
BS4, getting exact match between unclosed <br>
from bs4 import BeautifulSoup
html = '''<tbody id="plaintiff-body">
<tr>
<td><img id="plaimg0001" src="/CaseInformationOnline/images/minus.png&...
0
votes
3
answers
26
views
Beautifulsoup select an element based on the innerHTML with Python
I am writing a python script with Beautifulsoup to scrap the questions page on Stack Overflow. I only want to get all of the question titles along with the votes from the page.
For each of the ...
0
votes
0
answers
14
views
UnicodeEncodeError: 'charmap' codec can't encode characters in position 726-730: character maps to <undefined> [duplicate]
I have the following code in VSCode:
from bs4 import BeautifulSoup
import urllib.request
req = urllib.request.urlopen('https://www.ua-football.com/sport')
html = req.read()
soup = BeautifulSoup(html,...
0
votes
2
answers
22
views
Scrapping HLTM table from with beautifulsoup
I have been trying to do some web scrapping on this tables website (https://prosettings.net/cs-go-pro-settings-gear-list/) to practice some python skills however it does not return a full table but ...
0
votes
0
answers
17
views
Can't seem to scrape specific information from webpage?
I'm attempting to scrape some information for each item displayed on the following page:
https://www.finewineandgoodspirits.com/webapp/wcs/stores/servlet/CatalogSearchResultView?storeId=10051&...
-2
votes
1
answer
21
views
Need Wikipedia web scraper to continuously ask for user input
I need the below code to ask for user input again, after executing and showing results. I guess a while loop would be best but not sure how to do it as have BeautifulSoup and requests library in use.
...
-3
votes
0
answers
35
views
Code won't work. is this an issue with my logic or my loop? [closed]
I have a short script that compares two websites to see if they are similar. For some reason, when I process this code, it does not work.
For example, when I attempt to print out check_list at the end ...
-2
votes
0
answers
13
views
Problem scraping web page for different information
I'm having trouble scraping the elements on this web page e.g. the heading (h4).
https://www.jnjmedtech.com/en-US/search-company?search_api_fulltext=10439011
My code looks like this. Why do I get the ...
1
vote
2
answers
31
views
Extracting RegEx pattern across list excluding other html code
I've written a script to pull a list of available report url extensions page available for text extraction.
I've used parsing and BeautifulSoup to extract the reference area for the latest report ...
1
vote
1
answer
23
views
Using BeautifulSoup to get text and image urls from HTML string
class BlogApi(object):
def __init__(self):
json = "https://remaster.realmofthemadgod.com/index.php?rest_route=/wp/v2/posts/"
with urllib.request.urlopen(f"{rotmgjson}") as ...
0
votes
1
answer
28
views
Web Scraping a list of links from Tripadvisor
I'm trying to create a webscraper that will return a list of links to individual objects from the website example website.
The code I wrote takes the list of pages and returns the list of links to ...
0
votes
1
answer
19
views
Scraping table contents from coingecko using beautifulsoup4
This is the code/output I got by referencing here: scraping coins market with BeautifulSoup.
However, I cannot make this code run:
url = 'https://www.coingecko.com/en/coins/ethereum'
soup = ...
1
vote
1
answer
28
views
AttributeError: 'NoneType' object has no attribute 'text' in beautiful soup [closed]
My goal is to scrape the stock price of the company which user will enter in the input box, but I am getting this error.
When I'm running the code without input block it's running fine,
I don't know ...
-3
votes
0
answers
35
views
Scraping with beatiful soup, getting 'NoneType' error
I am working on an html parse using beautiful soup and scraping the website 'Trulia' and I had a code that worked last week and when I went to run it this week it returns the 'NoneType' error and ...
0
votes
1
answer
39
views
Is there a way to separate strings in HTML?
I'm trying to get the address of some companies from WSJ.com. However, I couldn't figure out a reliable way to separate the city and the state/province from the HTML page.
here's my code and output
...
1
vote
1
answer
17
views
Adding a tag around a certain word in XML
Let's say my xml looks like this:
<Dialog team="1">
<Participants>
<Person nickname="Leibniz"/>
<Person nickname="Clarke"/>
</Participants>
...
1
vote
4
answers
57
views
Beautiful Soup not working with requests.get
So I am a python beginner trying to scrape this website http://www.edwaittimes.ca/WaitTimes.aspx
that gives wait times for hospitals. Currently I am trying to print all the names of the hospitals.
My ...
1
vote
1
answer
14
views
Get the first regular link that is not located between brackets out of a paragraph using python (bs4)
I need to write a program in python using bs4 that shows me the path of one wikipedia site to another, for this I have to take the first link on the current wikipedia site that is located in the 'div',...
1
vote
1
answer
17
views
Simple tag replacement with BeautifulSoup
I have a very simple tag replacement problem that I'm trying to solve with BeatifulSoup's replace_with method but I'm having trouble understanding how this is supposed to work. I have the string '<...
-3
votes
0
answers
20
views
Beautifulsoup Error : 'bytes' object has no attribute 'find_all' [closed]
I have created a script months back using Beautifulsoup, which was working fine without any issue. 7 days back I started to get this error even on printing soup
Traceback (most recent call last):
...
1
vote
1
answer
41
views
How can I speed up parsing with aiohttp workers queue?
I need to collect 256 urls from the main page, 653 more of them, 11000 more of these 653. I collected 653, guy helped me here.Help me add an async to the worker_iso(q) function, which collects 11000 ...
0
votes
1
answer
46
views
Extract ticketcorner event seat prices from a venue map
I want to use beautifulsoup to extract data from a tooltipster that is hovering over a map. Once it finds a point with information, it has to extract three different pieces of information. If it doesn'...
2
votes
1
answer
51
views
How can I speed up the aiohttp parser bs4?
The task is to get data from the site. I have 800 URLs to request. But it takes a long time .I use aiohttp. At this stage, I have received links, by clicking on each of them, I also get a number of ...
0
votes
1
answer
18
views
Get a specific tag - BeautifulSoup
Below is the xml that I'm trying to parse.
<url>
<loc>https://www.houseofindya.com/aqua-chanderi-pleated-sharara-pants-177/iprdt</loc>
<image:image>
<image:loc&...
-2
votes
2
answers
50
views
having trouble in getting all the P tags of page
the article passages are divided into different divs tags
like in the image you can see it is written data-page-number="2"
just like that the data is divided into 8 different divs and the ...
0
votes
2
answers
37
views
neither find_all nor find works
I am trying to scrape the name of every favorites on the page of a user of our choice. but with this code I get the error "ResultSet object has no attribute 'find_all'" but if I try to use ...
-1
votes
1
answer
29
views
if class not found return none in beatifulSoup
I'm trying to get None if the class is not found in web scraping. For example, in some cases stage-codes.html#10_99 doesn't exist in HTML.
for st in soup.find_all("ul", {"class": &...
-1
votes
0
answers
12
views
getting NoSuchWindowException while scrapping twitter usernames using Selenium
I have been trying to scrape twitter usernames by going inside the followers page but the issue is if I leave my pc there after some time I get this exception and also If I look into the browser it ...
0
votes
1
answer
21
views
Handling this website which is redirecting to the same url with BeautifulSoup
So I'm scrapping this website: https://www.hepsiburada.com/apple-macbook-pro-m1-cip-8gb-256gb-ssd-macos-13-qhd-tasinabilir-bilgisayar-uzay-grisi-myd82tu-a-p-HBV0000130VNO-yorumlari
I'm scraping all ...
1
vote
1
answer
48
views
Scraping data with BeautifulSoup and Selenium
I am using BeautifulSoup and Selenium to extract web data (beautifulsoup to parse the HTML page and Selenium to click Next to get to the next list of items on the page).
What I need the code to do is:
...
0
votes
0
answers
14
views
Dynamic(with mouseover/coordinates) web scraping python unable to extract information
I'm trying to scrape the data that only appears on mouseover(selenium). It's a concert map and this is my entire code. I keep getting TypeError: 'ActionChains' object is not iterable
The idea would be ...
1
vote
2
answers
29
views
How to select and scrape specific texts out of a bunch <ul> and <li>?
I need to scrape "2015" and "09/09/2015" from the below link:
lacentrale.fr/auto-occasion-annonce-87102353714.html
But since there are many li and ul, I cant scrape the exact text. ...
0
votes
1
answer
22
views
Problem in fetching long URLs using BeautifulSoup
I am trying to fetch a URL from a webpage, here is how the URL looks in the Inspect section:
Here is how the URL looks in my python-code:
How can I get the actual URL without the ../../ part using ...
-2
votes
1
answer
23
views
How to ignore infobox when scraping title from Wikipedia anchor text?
I am trying to scrape the first 20 links on a Wikipedia page but I want to ignore the infobox on the right side. It has a 'table' tag. Here is what I have so far, any help would be greatly appreciated....
-2
votes
0
answers
16
views
Need href of all the Real Estate Projects - BeautifulSoup [closed]
link - https://www.immoweb.be/en/search/house/for-sale/limburg/province?countries=BE&page=1&orderBy=relevance
tried using selenium and beautiful soup but still cannot extract href of all the ...
-1
votes
0
answers
35
views
Scraping Wikipedia for information with Beautiful Soup
I managed to scrape wikipedia for names of US Presidents using Beautiful Soup. After which I converted them into dataframe.
names=[all the president's name]
wiki =[url of the president's wiki page]
...
-1
votes
0
answers
37
views
How to speed up python data parsing?
I have such a task - i need to parse the site in the form of a taxonomy and save to csv, that is, upload 24,000 links, that is, I uploaded 800 links to a file, reading line by line, collecting data, ...
0
votes
2
answers
28
views
How to convert Python's BeautifulSoup .text to uppercase?
I am attempting to webscrape the titles from articles on the BBC website and convert it to uppercase through the python .upper method. I'm using Python's BeautifulSoup library for this. This is my ...
1
vote
1
answer
45
views
Scrape and change data in date in BeautifulSoup
I am scraping data from different web pages and there are several dates in this data. The code allowing me to have the information that I want looks like this, I only put here the part concerning the ...
-1
votes
0
answers
20
views
How to take a specific text from a complicated source code using bs4?
I am trying to get the text of the code below,
<xmp:createdate>2020-12-31T17:54:35+08:00</xmp:createdate>
from the source code of this url(and the variations of it depending on the id of ...
-3
votes
0
answers
20
views
Need Help for Python Project [closed]
Project Description
There are two online supermarkets which provide price of various products.
You need to write a program to acquire, extract and compare the price from these two suppliers, for a ...
-1
votes
1
answer
28
views
'NoneType' object has no attribute 'text' when attempting to retrieve Table Data
People have asked similar questions like this 100 times before but none of the solutions are working to fix my issue! I have created a html doc that I am hosting off github that has a table on it! The ...
-1
votes
1
answer
60
views
How to fix Deprecation Warning: executable_path has been deprecated, please pass in a Service object
I am pretty new to coding and Python - The scraper starts off well and works, until at some point (after around 1 minute or so) it stops and hands out this error message.
DeprecationWarning: ...
1
vote
2
answers
36
views
How to seperate and replace specific scraped strings?
I want to clean up scraped text from <dd> and <dt>, output looks like this:
Įrengimas:': 'Dalinė apdailaNAUDINGA:Interjero dizaineriai', 'Ypatybės:': 'Nauja kanalizacijaNauja elektros ...
-4
votes
3
answers
43
views
python beautifulsoup how to get data as json format?
I want to get data as json format. Right now I am getting data as dictionary which little bit messy for me. Here is my code:
my_dict = {"job_title":[],"time_posted":[],"...
0
votes
1
answer
29
views
Selenium: get the url of a button when clicking without being redirected to the page
Scraping the page https://www.milanuncios.com/viviendas-en-eixample-esquerra-barcelona-barcelona/?demanda=n&fromSearch=1&orden=date, i try to get the next page without being redirected. The ...
-3
votes
0
answers
21
views
Scraping local HTML Table into SQL [closed]
I need some help scraping data from a locally HTML file (Table content) to a database (SQL).
The data is updated every 5 minutes, so if the "computername" row is already added to the DB it ...
0
votes
2
answers
27
views
Retrieve all names from html tags using BeautifulSoup
I managed to setup by Beautiful Soup and find the tags that I needed. How do I extract all the names in the tags?
tags = soup.find_all("a")
print(tags)
After running the above code, I got ...
-1
votes
0
answers
23
views
How to specify needed fields using Beautiful Soup and properly call upon website elements using HTML tags
I have been trying to create a web scraping program that will return the values of the Title, Company, and Location from job cards on Indeed. I finally am not returning error codes, however, I am only ...