177

I am trying to learn how to automatically fetch urls from a page. In the following code I am trying to get the title of the webpage:

import urllib.request
import re

url = "http://www.google.com"
regex = r'<title>(,+?)</title>'
pattern  = re.compile(regex)

with urllib.request.urlopen(url) as response:
   html = response.read()

title = re.findall(pattern, html)
print(title)

And I get this unexpected error:

Traceback (most recent call last):
  File "path\to\file\Crawler.py", line 11, in <module>
    title = re.findall(pattern, html)
  File "C:\Python33\lib\re.py", line 201, in findall
    return _compile(pattern, flags).findall(string)
TypeError: can't use a string pattern on a bytes-like object

What am I doing wrong?

1

3 Answers 3

256

You want to convert html (a byte-like object) into a string using .decode, e.g. html = response.read().decode('utf-8').

See Convert bytes to a Python String

2
  • 2
    This solved the error TypeError: cannot use a string pattern on a bytes-like object but then I got errors like UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb2 in position 1: invalid start byte. I fixed it by using .decode("utf-8", "ignore"): stackoverflow.com/questions/62170614/…
    – baptx
    Commented Aug 17, 2020 at 22:18
  • "ignore" ignores. If that is what you want, then all is good. However sometimes this kind of problem belies a deeper problem, e.g. that the thing you want to decode really isn't decodable or meant to be, e.g. compressed or encypted text. Or it might need some other encoding like utf-16 . Caveat emptor.
    – rocky
    Commented Aug 17, 2020 at 23:50
58

The problem is that your regex is a string, but html is bytes:

>>> type(html)
<class 'bytes'>

Since python doesn't know how those bytes are encoded, it throws an exception when you try to use a string regex on them.

You can either decode the bytes to a string:

html = html.decode('ISO-8859-1')  # encoding may vary!
title = re.findall(pattern, html)  # no more error

Or use a bytes regex:

regex = rb'<title>(,+?)</title>'
#        ^

In this particular context, you can get the encoding from the response headers:

with urllib.request.urlopen(url) as response:
    encoding = response.info().get_param('charset', 'utf8')
    html = response.read().decode(encoding)

See the urlopen documentation for more details.

0
0

Based upon last one, this was smimple to do when pdf read was done .

text = text.decode('ISO-8859-1') 

Thanks @Aran-fey

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.