1

I'm trying to save the page source to a file, so that I don't have to constantly re-run my code every time I want to test something.

I have:

html_source = driver.page_source
soup = BeautifulSoup(html_source, 'lxml') # added `lxml` only b/c I got a warning saying I should
soup = soup.prettify()
with open('pagesource.html', 'wb') as f_out:
    f_out.write(soup)

The error I get is:

UnicodeEncodeError: 'ascii' codec can't encode character u'\xab' in position 223871: ordinal not in range(128)

I also tried f_out.write(str(soup)), which didn't work.

How do I write the content to a file?

0

3 Answers 3

0

BeautifulSoup is for parsing Html and not grabbing it. If you can import urllib, try urlretrieve:

import urllib
urllib.urlretrieve("http://www.example.com/test.html", "test.txt")
1
  • I think this is not exactly what is happening here - the OP uses selenium to approach a dynamically loaded web-site, then prettifying and "fixing" with BeautifulSoup the HTML source code retrieved from the browser automated by selenium and saving the resulting HTML on disk.
    – alecxe
    Oct 7, 2015 at 23:48
0

This works for me:

import urllib2

html = urllib2.urlopen('http://www.example.com').read()

Now html contains the source code of that url.

  with open('web.html', 'w') as f:
      f.write(html)

You should now be able to open that with a browser.

4
  • 1
    I think this is not exactly what is happening here - the OP uses selenium to approach a dynamically loaded web-site, then prettifying and "fixing" with BeautifulSoup the HTML source code retrieved from the browser automated by selenium and saving the resulting HTML on disk.
    – alecxe
    Oct 7, 2015 at 23:49
  • @alecxe Couldn't the OP prettify and 'fix' after saving the source to a file? I thought the point was to get the source, and then modify it
    – Totem
    Oct 7, 2015 at 23:51
  • It's likely that the OP wants to save the prettified version of the complete rendered by a browser page and then do smth with that file..not sure, let's see what feedback we'll get from the OP. Thanks.
    – alecxe
    Oct 7, 2015 at 23:53
  • @alecxe Yeah, I want to save the prettified version. spot on. Oct 8, 2015 at 0:22
0

From bs4 documentation:

UnicodeEncodeError: 'charmap' codec can't encode character u'\xfoo' in position bar (or just about any other UnicodeEncodeError) - This is not a problem with Beautiful Soup. This problem shows up in two main situations. First, when you try to print a Unicode character that your console doesn’t know how to display. (See this page on the Python wiki for help.) Second, when you’re writing to a file and you pass in a Unicode character that’s not supported by your default encoding. In this case, the simplest solution is to explicitly encode the Unicode string into UTF-8 with u.encode("utf8").

I got the same error and solved it using:

soup = BeautifulSoup(page.content, 'html.parser', from_encoding="utf8")
with open(file_name_with_path, mode="w",  encoding="utf8") as code:
        code.write(str(soup2.prettify()))

You should avoid writing in binary mode. Try using mode="w" instead of mode="wb". Also you have to specify that your file is being written in utf8 encoding. Your error was not due to bs4 but due to incapability of the file writing process from accepting utf8 coding.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.