792

I am trying to urlencode this string before I submit.

queryString = 'eventName=' + evt.fields["eventName"] + '&' + 'eventDescription=' + evt.fields["eventDescription"]; 

16 Answers 16

1323

Python 2

What you're looking for is urllib.quote_plus:

safe_string = urllib.quote_plus('string_of_characters_like_these:$#@=?%^Q^$')

#Value: 'string_of_characters_like_these%3A%24%23%40%3D%3F%25%5EQ%5E%24'

Python 3

In Python 3, the urllib package has been broken into smaller components. You'll use urllib.parse.quote_plus (note the parse child module)

import urllib.parse
safe_string = urllib.parse.quote_plus(...)
9
  • 10
    Thanks! In my case however I need to put: import urllib.parse ... urllib.parse.quote_plus(query)
    – ivkremer
    Commented Sep 2, 2014 at 11:53
  • 3
    very good,but why does not used to Unicode?if the url string is Unicode,I must encode it to UTF-8.Is there any other way to do it? Commented Feb 6, 2015 at 6:55
  • 11
    This works great, but I couldn't access some online services (REST) until I added this parameter safe=';/?:@&=+$,'
    – rovyko
    Commented May 29, 2015 at 14:56
  • 1
    python3 -c "import urllib.parse, sys; print(urllib.parse.quote_plus(sys.argv[1])) "string to encode" for a one liner on the command line Commented May 8, 2019 at 12:19
  • 2
    @AmosJoshua I think you missed a double quote " just after the double round closing brackets )), it should be: python3 -c "import urllib.parse, sys; print(urllib.parse.quote_plus(sys.argv[1]))" "string to encode"
    – lgespee
    Commented Nov 2, 2021 at 19:04
735

You need to pass your parameters into urlencode() as either a mapping (dict), or a sequence of 2-tuples, like:

>>> import urllib
>>> f = { 'eventName' : 'myEvent', 'eventDescription' : 'cool event'}
>>> urllib.urlencode(f)
'eventName=myEvent&eventDescription=cool+event'

Python 3 or above

Use urllib.parse.urlencode:

>>> urllib.parse.urlencode(f)
eventName=myEvent&eventDescription=cool+event

Note that this does not do url encoding in the commonly used sense (look at the output). For that use urllib.parse.quote_plus.

14
  • 21
    "Note that the urllib.urlencode does not always do the trick. The problem is that some services care about the order of arguments, which gets lost when you create the dictionary. For such cases, urllib.quote_plus is better, as Ricky suggested."
    – Blairg23
    Commented Aug 17, 2015 at 21:35
  • 40
    Technically, that's a bug in the services, isn't it?
    – holdenweb
    Commented Aug 19, 2015 at 9:52
  • 12
    and how would one do this if you just want to make a string URL safe, without building a full query argument string? Commented Jun 7, 2016 at 2:14
  • 1
    @Mike'Pomax'Kamermans -- see e.g. stackoverflow.com/questions/12082314/… or Ricky's answer to this question.
    – bgporter
    Commented Jun 7, 2016 at 12:12
  • 1
    @bk0 it seems your method is only valid for dictionaries, and not strings.
    – JD Gamboa
    Commented Feb 2, 2018 at 22:23
100

Try requests instead of urllib and you don't need to bother with urlencode!

import requests
requests.get('http://youraddress.com', params=evt.fields)

EDIT:

If you need ordered name-value pairs or multiple values for a name then set params like so:

params=[('name1','value11'), ('name1','value12'), ('name2','value21'), ...]

instead of using a dictionary.

3
  • 8
    This does not address the issue of ordering the name value pairs, also this requires the permission to install external libraries which might not be doable for the project.
    – dreftymac
    Commented Sep 6, 2013 at 2:10
  • 1
    I posted the minimal code that would work for the OP. The OP did not request ordered pairs however it is also doable, see my update. Commented Sep 10, 2013 at 8:02
  • @dreftymac: this does address ordering (although it was not part of the question), please read my updated answer. Commented Nov 25, 2013 at 20:26
42

Context

  • Python (version 2.7.2 )

Problem

  • You want to generate a urlencoded query string.
  • You have a dictionary or object containing the name-value pairs.
  • You want to be able to control the output ordering of the name-value pairs.

Solution

  • urllib.urlencode
  • urllib.quote_plus

Pitfalls

Example

The following is a complete solution, including how to deal with some pitfalls.

### ********************
## init python (version 2.7.2 )
import urllib

### ********************
## first setup a dictionary of name-value pairs
dict_name_value_pairs = {
  "bravo"   : "True != False",
  "alpha"   : "http://www.example.com",
  "charlie" : "hello world",
  "delta"   : "1234567 !@#$%^&*",
  "echo"    : "[email protected]",
  }

### ********************
## setup an exact ordering for the name-value pairs
ary_ordered_names = []
ary_ordered_names.append('alpha')
ary_ordered_names.append('bravo')
ary_ordered_names.append('charlie')
ary_ordered_names.append('delta')
ary_ordered_names.append('echo')

### ********************
## show the output results
if('NO we DO NOT care about the ordering of name-value pairs'):
  queryString  = urllib.urlencode(dict_name_value_pairs)
  print queryString 
  """
  echo=user%40example.com&bravo=True+%21%3D+False&delta=1234567+%21%40%23%24%25%5E%26%2A&charlie=hello+world&alpha=http%3A%2F%2Fwww.example.com
  """

if('YES we DO care about the ordering of name-value pairs'):
  queryString  = "&".join( [ item+'='+urllib.quote_plus(dict_name_value_pairs[item]) for item in ary_ordered_names ] )
  print queryString
  """
  alpha=http%3A%2F%2Fwww.example.com&bravo=True+%21%3D+False&charlie=hello+world&delta=1234567+%21%40%23%24%25%5E%26%2A&echo=user%40example.com
  """ 
0
37

Python 3:

urllib.parse.quote_plus(string, safe='', encoding=None, errors=None)

1
  • 30
    I prefer urllib.parse.quote() myself since it uses %20 rather than +.
    – binki
    Commented Sep 29, 2015 at 20:43
28

Try this:

urllib.pathname2url(stringToURLEncode)

urlencode won't work because it only works on dictionaries. quote_plus didn't produce the correct output.

2
  • 1
    That's really helpful! In my case, I only have a portion of string that I want to URL-encode, for example I want to transform my string to my%20string. Your solution works like a charm for that!
    – TanguyP
    Commented Feb 14, 2016 at 11:27
  • 2
    In Python 3. It is now urllib.request.pathname2url Commented Oct 29, 2021 at 0:52
22

Note that the urllib.urlencode does not always do the trick. The problem is that some services care about the order of arguments, which gets lost when you create the dictionary. For such cases, urllib.quote_plus is better, as Ricky suggested.

1
  • 2
    It works fine and preserves order if you pass a list of tuples: >>> import urllib >>> urllib.urlencode([('name', 'brandon'), ('uid', 1000)]) 'name=brandon&uid=1000' Commented Mar 16, 2017 at 21:07
15

In Python 3, this worked with me

import urllib

urllib.parse.quote(query)
2
  • It worked for me to encode a string as part of the whole url Commented Jan 28, 2022 at 8:19
  • I had to do import urllib.parse, otherwise I get "AttributeError: module 'urllib' has no attribute 'parse'". Commented Nov 28, 2023 at 16:41
12
import urllib.parse
query = 'Hellö Wörld@Python'
urllib.parse.quote(query) # returns Hell%C3%B6%20W%C3%B6rld%40Python
2
  • 1
    urllib.parse.quote is already mentioned in this earlier answer. Commented Nov 19, 2021 at 3:38
  • @GinoMempin The problem with the other answer is that says import urllib when it should be urllib.parse. Yes, they could also have suggested an edit, but as it is now, this answer is the most correct of the two. Commented Nov 28, 2023 at 16:46
6

for future references (ex: for python3)

>>> import urllib.request as req
>>> query = 'eventName=theEvent&eventDescription=testDesc'
>>> req.pathname2url(query)
>>> 'eventName%3DtheEvent%26eventDescription%3DtestDesc'
2
  • 1
    usually you only want to url encode the values, what you have done here would make an invalid GET query Commented Feb 24, 2015 at 14:38
  • 1
    The output for 'c:/2 < 3' on Windows is '///C://2%20%3C%203'. I want something that would just output 'c:/2%20%3C%203'.
    – binki
    Commented Sep 29, 2015 at 20:42
5

If the urllib.parse.urlencode( ) is giving you errors , then Try the urllib3 module .

The syntax is as follows :

import urllib3
urllib3.request.urlencode({"user" : "john" }) 
5

For use in scripts/programs which need to support both python 2 and 3, the six module provides quote and urlencode functions:

>>> from six.moves.urllib.parse import urlencode, quote
>>> data = {'some': 'query', 'for': 'encoding'}
>>> urlencode(data)
'some=query&for=encoding'
>>> url = '/some/url/with spaces and %;!<>&'
>>> quote(url)
'/some/url/with%20spaces%20and%20%25%3B%21%3C%3E%26'
1

Another thing that might not have been mentioned already is that urllib.urlencode() will encode empty values in the dictionary as the string None instead of having that parameter as absent. I don't know if this is typically desired or not, but does not fit my use case, hence I have to use quote_plus.

0

For Python 3 urllib3 works properly, you can use as follow as per its official docs :

import urllib3

http = urllib3.PoolManager()
response = http.request(
     'GET',
     'https://api.prylabs.net/eth/v1alpha1/beacon/attestations',
     fields={  # here fields are the query params
          'epoch': 1234,
          'pageSize': pageSize 
      } 
 )
response = attestations.data.decode('UTF-8')
0

If you don't want to use urllib.

https://github.com/wayne931121/Python_URL_Decode

URL_RFC_3986 = {
"!": "%21", "#": "%23", "$": "%24", "&": "%26", "'": "%27", "(": "%28", ")": "%29", "*": "%2A", "+": "%2B", 
",": "%2C", "/": "%2F", ":": "%3A", ";": "%3B", "=": "%3D", "?": "%3F", "@": "%40", "[": "%5B", "]": "%5D",
}

def url_encoder(b):
    # https://zh.wikipedia.org/wiki/%E7%99%BE%E5%88%86%E5%8F%B7%E7%BC%96%E7%A0%81
    if type(b)==bytes:
        b = b.decode(encoding="utf-8") #byte can't insert many utf8 charaters
    result = bytearray() #bytearray: rw, bytes: read-only
    for i in b:
        if i in URL_RFC_3986:
            for j in URL_RFC_3986[i]:
                result.append(ord(j))
            continue
        i = bytes(i, encoding="utf-8")
        if len(i)==1:
            result.append(ord(i))
        else:
            for c in i:
                c = hex(c)[2:].upper()
                result.append(ord("%"))
                result.append(ord(c[0:1]))
                result.append(ord(c[1:2]))
    result = result.decode(encoding="ascii")
    return result

#print(url_encoder("我好棒==%%0.0:)")) ==> '%E6%88%91%E5%A5%BD%E6%A3%92%3D%3D%%0.0%3A%29'
2
  • I don't know why this was downvoted. I found it to be just the job for Micropython, which lacks urllib. I had to take out the keyword arguments (they aren't supported on the ESP8266 version) and adjust the translation table for my needs, but then it worked perfectly!
    – Graham
    Commented Jun 18 at 10:06
  • Mmh shouldn't be escaped also % ?
    – Massimo
    Commented Jun 22 at 17:03
0

urllib parse wasn't quite sufficient in my case.

Two additions were needed:

  1. use safe kwarg to unmark / as a safe character (urllib.parse.quote(string, safe=''))

  2. use .replace() for remaining special characters - in particular, note that javascript's encodeURIComponent and C# Uri.EscapeDataString behave differently. For working with a C# backend that expects EscapeDataStrings, then the following all need to be additionally replaced: _-.~

Ultimately I have something like this

import urllib

def aggressive_urlencode(inp: str) -> str:
    return (
        urllib.parse.quote(inp, safe='')
        .replace('-', '%2D')
        .replace('.', '%2E')
        .replace('_', '%5F')
        .replace('~', '%7E')
    )

Not the answer you're looking for? Browse other questions tagged or ask your own question.