Questions tagged [beautifulsoup]

Beautiful Soup is a Python package for parsing HTML/XML. The latest version of this package is version 4, imported as bs4.

Filter by
Sorted by
Tagged with
0 votes
0 answers
25 views

Elements missing in saved html from playwright

Im trying to scrape the prices from buff.163 but i cant find them in the html that was saved. My code looks like this: from playwright.sync_api import sync_playwright with sync_playwright() as p: ...
BlackPhantom's user avatar
0 votes
2 answers
58 views

extract value from span

I want to extract a snow depth value from a weather site to a dataframe. (https://www.yr.no/nb/sn%C3%B8dybder/NO-46/Norge/Vestland) Specifically the snow depth for the Jordalen - Nåsen area. Screen ...
slabbe's user avatar
  • 1
0 votes
1 answer
19 views

How to remove a tag from an element of BeautifulSoup?

I have a page like this: ... <div class="myclass"> <p> text 1 to keep<span>text 1 to remove</span>and keep this too. </p> <p> text 2 to keep&...
MarcoS's user avatar
  • 17.4k
1 vote
1 answer
22 views

how to get anchors href value and nested page anchors href value in using concurrent.futures?

I am trying to fetch the anchor tag href value and nested urls with concurrent.futures? I want to reducing the time to fetch the urls using below code. it's taking time to fetch the URLs using below ...
Sam's user avatar
  • 843
1 vote
1 answer
30 views

Identify text with span elements using Python BeautifulSoup

I have html text that I am trying to clean up using "soup". However, I need to not only identify which text segments are contained by certain span elements of the class='highlight', but also ...
psychcoder's user avatar
-3 votes
2 answers
44 views

TypeError: write() argument must be str, not Tag [duplicate]

Here is my code: html = open("meetup.html", "r") from bs4 import BeautifulSoup soup = BeautifulSoup(html, 'html.parser') file = open('myfile.txt', 'w') for elem in soup.find_all(...
Ooker's user avatar
  • 2,178
2 votes
1 answer
42 views

How to extract text from HTML with BeautifulSoup?

from bs4 import BeautifulSoup import requests website = 'https://www.klavkarr.com/data-trouble-code-obd2.php?dtc=p0000-p0299#dtc' result = requests.get(website) content = result.text soup = ...
Emiliano Cerasani's user avatar
0 votes
1 answer
27 views

Beautifulsoup: find the location of a string in the soup

I am trying to scrap data from a web page generated by a search engine. For this purpose, I study the data structure of the document using a sample query and the output result. The output contains a ...
freude's user avatar
  • 3,672
1 vote
1 answer
33 views

how to modify a BeautifulSoup object

Let's presume I have the following piece of code text = '<link href="link">' souped = BeautifulSoup(text, 'html.parser') tags = souped.find_all() link_modified_tags = ResultSet() ...
Dimon Agon's user avatar
2 votes
1 answer
29 views

Webscraping Entry id from the Uniprot database search result

Dear Stack Overflow community, I'm working on a project where I need to extract entry IDs from the UniProt website's search results. Specifically, I'm looking to retrieve IDs like 'O00622' from HTML ...
SH Kim's user avatar
  • 21
0 votes
0 answers
18 views

Enable cookies and javascript with BS4

I have coded a scraper using BS4 to get some stats off a betting website. The scraper was working but now I am getting this error saying I need to enable JS and cookies to continue: <!DOCTYPE html&...
Matthew Trickey-Trehearn's user avatar
1 vote
1 answer
49 views

Trouble getting an html table from an aspx website

I am trying to extract the table from this website: https://serviciosede.mineco.gob.es/indeco/reports/verSerieGraf.aspx/?codigo=230400&frec=-1 The table starts with column headers "Fecha"...
encore10's user avatar
-3 votes
0 answers
16 views

how to get discounted price using meta property tag in beautifulsoup python? [closed]

My code is: price = soup.find("meta", attrs={"property": "product:price:amount"}).get("content") I want to get discounted price from a webpage like below: ...
Martin's user avatar
  • 1
-1 votes
0 answers
42 views

While converting scraped data into csv file hotel price and other things showing in special word [closed]

import pandas as pd import requests from bs4 import BeautifulSoup Hotel = [] Description = [] Prices = [] for page_number in range(1, 16): url = f"https://www.airbnb.co.in/s/Jaipur--India/...
Rabin Sunuwar's user avatar
1 vote
2 answers
32 views

How to scrape section of HTML and extract text?

I am trying to extract the string 2 and a half stars, 4688 Ratings in python from the below HTML element. I used this command and got this set of data. I can't figure out how to extract this portion ...
JMHA1997's user avatar
-3 votes
0 answers
48 views

Is there any way to use BeautifulSoup to webscrape from Bible.com? [closed]

I'm trying to webscrape a verse from a specific version of the Bible in Turkish from Bible.com, the Kutsal Kitap Yeni Çeviri 2001, 2008. I'm able to use Beautiful Soup to webscrape from most websites, ...
Charlie Hou's user avatar
0 votes
2 answers
44 views

Basic webscraping-but I cannot solve this problem

I want to practice webscrapping and follow some video. That video said that: import requests from bs4 import BeautifulSoup url = "https://comic.naver.com/webtoon/list?titleId=811721&tab=wed&...
SH Kim's user avatar
  • 21
0 votes
1 answer
45 views

Web Scraping returns T&C link

This URL has some CSV data files I'd like to use for analysis under the folder 'usage-stats'. However, I'm unable to scrape data from this page. When I try using the below code, I can only see the ...
a_jelly_fish's user avatar
-1 votes
0 answers
28 views

Web Scraping - Cant find data on a html page

I am trying to scrape the following web page 'https://www.racingpost.com/profile/horse/3763274/kerkyra-island/form' In particualar I would like to scrape the data in the red box in the photo attached ...
Sam Kirwan's user avatar
-2 votes
0 answers
42 views

CSS properties supported by Microsoft Word [closed]

I'm converting a .html file to .docx format, and I want to know if there are some properties like <thead> with display: table-header-group to repeat the header of a table in each page. I want to ...
devStack's user avatar
-2 votes
1 answer
58 views

How to find div with specific class with bs4

I'm trying to scrape a specific class of divs from an HTML document using bs4. However, when I use the find_all() method, I do not get back the divs that I want despite the fact that I can see those ...
Rohan Sheth's user avatar
-1 votes
1 answer
32 views

Trouble Scraping a Page Full of JavaScript Code Using Python and Selenium [duplicate]

I am currently working on a web scraping project using Python and Selenium, and I'm encountering difficulties with a particular webpage that seems to be heavily reliant on JavaScript. When I inspect ...
Mohamed Azzam's user avatar
-1 votes
1 answer
106 views
+50

Unable to load JavaScript and got pyppeteer error from webpage with requests

I'm trying to scrape, after loggin, this webpage: https://saywow.me If I use only BeautifulSoup and requests I get Please enable JavaScript to continue using this application. error page So, I decided ...
Fabix's user avatar
  • 269
-1 votes
0 answers
14 views

Getting the contents of a specific table using BeautifulSoup

I am trying to get the content in the table on this website (data.ers.usda.gov/reports.aspx?ID=17828); however, I am failing to locate the table so I can proceed. I have tried numerous code but none ...
Grace's user avatar
  • 1
0 votes
0 answers
20 views

Beautiful Soup returning XX instead of actual values [duplicate]

I am trying to Webscrape a website using requests and beautiful soup. But I am getting 'XXX' instead of the text within the tags. code: url = 'https://www.discover.com/student-loans/' req = opener....
Aditi's user avatar
  • 1
-1 votes
1 answer
45 views

Web Scraping, load more data into a HTML

I am looking to scrape data from the following web page https://www.racingpost.com/bloodstock/sales/catalogues/5/2023-12-04 I am using beatiful soup and have requested the html as seen below. The soup ...
Sam Kirwan's user avatar
-1 votes
0 answers
19 views

Can't upload media to wordpress using Python script and wordpress_xmlrpc

Pretty new to this so please keep it simple! Basically I have data I have scraped with a Python script from one URL which includes a title, description, video and images. I have another script which ...
Spaceshanty's user avatar
-2 votes
0 answers
32 views

How do i make this load more images? (Python) [closed]

I have this function I've written to search for a specific word on google images, and then acquire all the urls for the images it detects (except the firts one, which is always the google logo): def ...
Marco Antonio Pelonero's user avatar
-2 votes
0 answers
30 views

Need help to download images by searching for images on that website [closed]

So I am trying to webscrape images from the PIER digital library of images https://peir.path.uab.edu/library/search.php. (It is a public library). But I seem to encounter a problem. So I have a list ...
Aditya D's user avatar
1 vote
2 answers
46 views

Unable to fetch list from angular website using beautiful soup

I am trying to parse nptel website (url) to fetch all courses listed on it. Each course correspond to app-courses-card tag. The DOM looks like this: I am unable to get hold of these tags with ...
MsA's user avatar
  • 2,663
0 votes
2 answers
65 views

How to find main price and discounted price in a webpage using selenium and python?

I am trying to find a way to find main price and also discounted price in a webpage but I can get just one of them and I need a good pattern or method to extract all price and discounted prices from ...
Alireza Mirhabibi - IRAN's user avatar
0 votes
1 answer
48 views

Python Beatifulsoup only scrapes the 1st page of Amazon reviews, someone knows how to extract all ideally without a proxy?

I am using Beautifulsoup on Python to try to scrape all Amazon reviews for a product, but it only extracts the 1st page (9 reviews). This code seem to have worked in the past for other users, maybe I ...
Alfredo N.'s user avatar
1 vote
1 answer
57 views

how to extract table from a webpage using its id?

I was trying to gather some data for a data analysis project concerning italian football. Unfortunately I can't extract one specific table from the web page, even though the id is correct. The code is ...
Walrus's user avatar
  • 13
-2 votes
0 answers
19 views

Parsing (and updating) paragraph with embedded tags [closed]

I'm trying to parse and modify the words in a paragraph (that may have other tags (i.e. formatting)) with a XHTML file. For example, XHTML=b"""<?xml version="1.0" encoding=&...
Richard Offer's user avatar
0 votes
1 answer
82 views

Error using BeautifulSoup code to retrieve info from cooking website. Issues parsing HTML

I am attempting to create a program that will bring back the Title, URL, Ingredients, and Recipe Instructions from recipes listed in the allrecipes.com sitemap. However, I'm missing something for the ...
DeskPrisoner's user avatar
-3 votes
0 answers
29 views

Problems with parsing hash [closed]

There is a website https://mrush.mobi/welcome I want to create a bot for the site, but there is a problem with the hash attribute Tell me how to get the correct hash? image I tried to write something ...
Денис Остап'юк's user avatar
0 votes
1 answer
42 views

How Can I Extract a Value from HTML?

I am building automation to deal with email alerts that we receive. The final step is to be able to extract the username involved in the alert and from research it looked like it should be fairly easy ...
Bronson888's user avatar
-2 votes
1 answer
31 views

How to get the all anchor tags current page and nested pages url? [closed]

I want to get the attribute value href in a anchor tags current page and nested from the URL https://www.tradeindia.com/ pages which is matches the value .*([0-9]+).html in href attribute value. Lile ...
Sam's user avatar
  • 843
1 vote
2 answers
45 views

Getting allways the same values when iterating using Selenium?

i try to collect some data using the following code: import time import os from bs4 import BeautifulSoup from selenium import webdriver from selenium.webdriver.chrome.options import Options from ...
Rapid1898's user avatar
  • 963
-2 votes
0 answers
39 views

'NoneType' object has no attribute 'contents' [closed]

problem: using .replace(",", "") method with .contents method of Beautiful-Soup to convert 103,121 to integer 103121 return using function alone doesnt give error but when be used ...
MohamedHany's user avatar
0 votes
2 answers
32 views

List of all static-ralated tags and elements, and how to get all static from a beautifulsoup

I working on a beautifulsoup finding of all elements that are related to static. What is the list of all static-related tags and elements? Is there any other way apart from finding by multiple ...
Dimon Agon's user avatar
1 vote
3 answers
54 views

How to extract only the first items from several span tags using Beautiful Soup?

I only began to learn Beautiful Soup recently. I'm stuck trying to extract only the first items from a part of the html code I recently scraped from a site. I simplified the original html code as ...
edoft's user avatar
  • 11
-1 votes
0 answers
31 views

My scraper search app is returning 400 Bad Request and I am unsure why

I'm trying to create a search engine which returns keyword results from findaphd.com. This is my Python code and I am unsure what I'm doing wrong. I'm quite new to coding so apologies if this is a ...
salamandrine's user avatar
0 votes
3 answers
105 views

How to parse xml to pandas dataframe using beatiful soup

I have: xml_data = """<address_details sequence_number="1" match_indicator="L"> <address_input id="ADI" address_details="street address ...
frank's user avatar
  • 3,132
-2 votes
0 answers
29 views

The content of scraped div is empty (Dynamic JS script) [closed]

Could someone please help me scraping information from transfermarkt? This is an example url: https://www.transfermarkt.com/georginio-wijnaldum/transfers/spieler/49499 This is how it looks in DevTools:...
ThijsTUE's user avatar
-1 votes
0 answers
69 views

Traceback (most recent call last), TypeError: 'NoneType' object is not callable

Project was to scrap the product information from Flipkart like Product Name, Price, Rating, Specifications. All extracted data stored in data frame by using pandas and then export to CSV format. When ...
Laxmi Narayana Pattanayak's user avatar
-1 votes
1 answer
35 views

Is it possible to use selenium to input to a website, then beautiful soup to parse the HTML info, then output to a desktop tool? [closed]

I am working on a weather tool and I need to be able to input a city name to a website, once that website give me the informaiton for the citys historical weather information I need to be able to pull ...
jessica gonzales's user avatar
-1 votes
1 answer
41 views

How do I change a particular part of a XML which has been read into Python and then create a new XML with this change? [duplicate]

So I am using BeautifulSoup to read and parse an XML below: from bs4 import BeautifulSoup with open(r"....TestXML.xml", 'r') as f: data = f.read() bs_data = BeautifulSoup(data, "...
Mark's user avatar
  • 33
-4 votes
1 answer
83 views

How to web scrape a number with python

I am trying to web scrape a simple number from this website I tried this code but get an empty variable: import requests from bs4 import BeautifulSoup url = "https://www.quantalys.com/Recherche?...
jacques's user avatar
  • 53
-1 votes
1 answer
41 views

Trip Advisor Rating count returns 'None' - Python Webscraping

I am tyring to scrape the Trip-Adivors number of reviews in Yellowpages. Here is the complete code from start to bottom. import requests from bs4 import BeautifulSoup import pandas as pd import time ...
Kbrosal's user avatar

1
2 3 4 5
650