0

I am trying to make my code print all the plain text from a website. This is my code:

import requests
import json
response = requests.get("https://example.com")
json_data = json.loads(response.text)
print(str(json_data))

For example: if I enter https://example.com, I want to make the program write

Example Domain

This domain is for use in illustrative examples in documents. You may use this domain in literature without prior coordination or asking for permission.

More information...

But I get this error message:

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    json_data = json.loads(response.text)
  File "/usr/lib/python3.8/json/__init__.py", line 357, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.8/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.8/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

How can I fix this?

1
  • Why are you calling json.loads()? Website responses typically aren't in json format. Jan 15, 2022 at 20:56

1 Answer 1

0
from urllib.request import urlopen
from bs4 import BeautifulSoup

url = "https://example.com"
html = urlopen(url).read()
soup = BeautifulSoup(html, features="html.parser")
# delete unwanted elements
for script in soup(["script", "style"]):
    script.extract()  

# get actual text
text = soup.get_text()


lines = (line.strip() for line in text.splitlines())
chunks = (phrase.strip() for line in lines for phrase in line.split("  "))
text = '\n'.join(chunk for chunk in chunks if chunk)

print(text)

don't forget to first pip install beautifulsoup4

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.