392

If I do

url = "http://example.com?p=" + urllib.quote(query)
  1. It doesn't encode / to %2F (breaks OAuth normalization)
  2. It doesn't handle Unicode (it throws an exception)

Is there a better library?

2
  • 5
    These are not URL parameters, FYI. You should clarify. Sep 7, 2018 at 18:53
  • What is the language-agnostic canonical Stack Overflow question? (That is, only covering the encoding, not how it is achieved.) Nov 27, 2022 at 21:54

6 Answers 6

507

Python 2

From the documentation:

urllib.quote(string[, safe])

Replace special characters in string using the %xx escape. Letters, digits, and the characters '_.-' are never quoted. By default, this function is intended for quoting the path section of the URL.The optional safe parameter specifies additional characters that should not be quoted — its default value is '/'

That means passing '' for safe will solve your first issue:

>>> urllib.quote('/test')
'/test'
>>> urllib.quote('/test', safe='')
'%2Ftest'

About the second issue, there is a bug report about it. Apparently it was fixed in Python 3. You can workaround it by encoding as UTF-8 like this:

>>> query = urllib.quote(u"Müller".encode('utf8'))
>>> print urllib.unquote(query).decode('utf8')
Müller

By the way, have a look at urlencode.

Python 3

In Python 3, the function quote has been moved to urllib.parse:

>>> import urllib.parse
>>> print(urllib.parse.quote("Müller".encode('utf8')))
M%C3%BCller
>>> print(urllib.parse.unquote("M%C3%BCller"))
Müller
8
  • 2
    Thanks you, both worked great. urlencode just calls quoteplus many times in a loop, which isn't the correct normalization for my task (oauth). Nov 8, 2009 at 9:14
  • 8
    the spec: rfc 2396 defines these as reserved reserved = ";" | "/" | "?" | ":" | "@" | "&" | "=" | "+" | "$" | "," Which is what urllib.quote is dealing with. Sep 23, 2015 at 17:42
  • 8
    urllib.parse.quote docs Dec 16, 2016 at 10:50
  • 1
    if you wanna retain the colon from http: , do urllib.parse.quote('http://example.com/some path/').replace('%3A', ':') May 9, 2019 at 7:27
  • 3
    @chrizonline Just use urllib.parse.quote(url, safe=':/'). Even better, encode some path, then join strings. This is Python, not PHP. Dec 23, 2021 at 9:26
204

In Python 3, urllib.quote has been moved to urllib.parse.quote, and it does handle Unicode by default.

>>> from urllib.parse import quote
>>> quote('/test')
'/test'
>>> quote('/test', safe='')
'%2Ftest'
>>> quote('/El Niño/')
'/El%20Ni%C3%B1o/'
3
  • 2
    The name quote is rather vague as a global. It might be nicer to use something like urlencode: from urllib.parse import quote as urlencode.
    – Luc
    Mar 5, 2019 at 16:35
  • 4
    Note that there is a function named urlencode in urllib.parse already that does something completely different, so you'd be better off picking another name or risk seriously confusing future readers of your code. Apr 2, 2020 at 2:41
  • 1
    (style suggestion: @Luc i agree that quote is "rather vague". rather than rename the variable/object to something else you can leave the name fully qualified as urllib.parse.quote. leaving it fully qualified does two things: takes a little extra time typing and saves time reading and maintaining the code. ) Jan 24 at 14:07
63

I think module requests is much better. It's based on urllib3.

You can try this:

>>> from requests.utils import quote
>>> quote('/test')
'/test'
>>> quote('/test', safe='')
'%2Ftest'

My answer is similar to Paolo's answer.

3
  • 8
    requests.utils.quote is link to python quote. See request sources.
    – Cjkjvfnby
    Aug 5, 2015 at 14:11
  • 23
    requests.utils.quote is a thin compatibility wrapper to urllib.quote for python 2 and urllib.parse.quote for python 3 Sep 23, 2015 at 17:30
  • without reading the comments, this is creating confusion...
    – PythoNic
    Jun 4, 2022 at 14:46
15

If you're using Django, you can use urlquote:

>>> from django.utils.http import urlquote
>>> urlquote(u"Müller")
u'M%C3%BCller'

Note that changes to Python mean that this is now a legacy wrapper. From the Django 2.1 source code for django.utils.http:

A legacy compatibility wrapper to Python's urllib.parse.quote() function.
(was used for unicode handling on Python 2)
1
  • it's deprecated from Django 3.0+
    – mosi_kha
    Nov 27, 2021 at 12:13
5

It is better to use urlencode here. There isn't much difference for a single parameter, but, IMHO, it makes the code clearer. (It looks confusing to see a function quote_plus! - especially those coming from other languages.)

In [21]: query='lskdfj/sdfkjdf/ksdfj skfj'

In [22]: val=34

In [23]: from urllib.parse import urlencode

In [24]: encoded = urlencode(dict(p=query,val=val))

In [25]: print(f"http://example.com?{encoded}")
http://example.com?p=lskdfj%2Fsdfkjdf%2Fksdfj+skfj&val=34

Documentation

1

An alternative method using furl:

import furl

url = "https://httpbin.org/get?hello,world"
print(url)
url = furl.furl(url).url
print(url)

Output:

https://httpbin.org/get?hello,world
https://httpbin.org/get?hello%2Cworld
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.