179

Suppose I was given a URL.
It might already have GET parameters (e.g. http://example.com/search?q=question) or it might not (e.g. http://example.com/).

And now I need to add some parameters to it like {'lang':'en','tag':'python'}. In the first case I'm going to have http://example.com/search?q=question&lang=en&tag=python and in the second — http://example.com/search?lang=en&tag=python.

Is there any standard way to do this?

0

15 Answers 15

232

There are a couple of quirks with the urllib and urlparse modules. Here's a working example:

try:
    import urlparse
    from urllib import urlencode
except: # For Python 3
    import urllib.parse as urlparse
    from urllib.parse import urlencode

url = "http://stackoverflow.com/search?q=question"
params = {'lang':'en','tag':'python'}

url_parts = list(urlparse.urlparse(url))
query = dict(urlparse.parse_qsl(url_parts[4]))
query.update(params)

url_parts[4] = urlencode(query)

print(urlparse.urlunparse(url_parts))

ParseResult, the result of urlparse(), is read-only and we need to convert it to a list before we can attempt to modify its data.

8
  • 19
    You probably want to use urlparse.parse_qs instead of parse_qsl. The latter returns a list whereas you want a dict. See docs.python.org/library/urlparse.html#urlparse.parse_qs. Jun 6, 2012 at 9:01
  • 12
    @florian : At least in python 2.7 you then need to call urlencode as urllib.urlencode(query, doseq=True). Otherwise, parameters that existed in the original url are not preserved correctly (because they are returned as tuples from @parse_qs@
    – rluba
    Sep 11, 2012 at 10:17
  • 5
    I've rewritten this to work in Python 3 as well. Code here.
    – duality_
    Jan 22, 2016 at 11:48
  • 16
    The results of urlparse() and urlsplit() are actually namedtuple instances. Thus you can assign them directly to a variable and use url_parts = url_parts._replace(query = …) to update it. Apr 5, 2016 at 14:08
  • 6
    Caution - this implementation removes repeated query parameters that some RESTful services use. With a little modification this can be fixed. query = urlparse.parse_qsl(url_parts[4]) query += params.items() But then if you want to replace exiting query params using dict, takes a little more.
    – ombre42
    Feb 28, 2017 at 16:14
89

Outsource it to the battle tested requests library.

This is how I will do it:

from requests.models import PreparedRequest
url = 'http://example.com/search?q=question'
params = {'lang':'en','tag':'python'}
req = PreparedRequest()
req.prepare_url(url, params)
print(req.url)
2
  • it is from requests packages? Apr 29, 2021 at 4:23
  • @perymerdeka yes Jun 18, 2021 at 6:20
72

Why

I've been not satisfied with all the solutions on this page (come on, where is our favorite copy-paste thing?) so I wrote my own based on answers here. It tries to be complete and more Pythonic. I've added a handler for dict and bool values in arguments to be more consumer-side (JS) friendly, but they are yet optional, you can drop them.

How it works

Test 1: Adding new arguments, handling Arrays and Bool values:

url = 'http://stackoverflow.com/test'
new_params = {'answers': False, 'data': ['some','values']}

add_url_params(url, new_params) == \
    'http://stackoverflow.com/test?data=some&data=values&answers=false'

Test 2: Rewriting existing args, handling DICT values:

url = 'http://stackoverflow.com/test/?question=false'
new_params = {'question': {'__X__':'__Y__'}}

add_url_params(url, new_params) == \
    'http://stackoverflow.com/test/?question=%7B%22__X__%22%3A+%22__Y__%22%7D'

Talk is cheap. Show me the code.

Code itself. I've tried to describe it in details:

from json import dumps

try:
    from urllib import urlencode, unquote
    from urlparse import urlparse, parse_qsl, ParseResult
except ImportError:
    # Python 3 fallback
    from urllib.parse import (
        urlencode, unquote, urlparse, parse_qsl, ParseResult
    )


def add_url_params(url, params):
    """ Add GET params to provided URL being aware of existing.

    :param url: string of target URL
    :param params: dict containing requested params to be added
    :return: string with updated URL
    
    >> url = 'https://stackoverflow.com/test?answers=true'
    >> new_params = {'answers': False, 'data': ['some','values']}
    >> add_url_params(url, new_params)
    'https://stackoverflow.com/test?data=some&data=values&answers=false'
    """
    # Unquoting URL first so we don't lose existing args
    url = unquote(url)
    # Extracting url info
    parsed_url = urlparse(url)
    # Extracting URL arguments from parsed URL
    get_args = parsed_url.query
    # Converting URL arguments to dict
    parsed_get_args = dict(parse_qsl(get_args))
    # Merging URL arguments dict with new params
    parsed_get_args.update(params)

    # Bool and Dict values should be converted to json-friendly values
    # you may throw this part away if you don't like it :)
    parsed_get_args.update(
        {k: dumps(v) for k, v in parsed_get_args.items()
         if isinstance(v, (bool, dict))}
    )

    # Converting URL argument to proper query string
    encoded_get_args = urlencode(parsed_get_args, doseq=True)
    # Creating new parsed result object based on provided with new
    # URL arguments. Same thing happens inside urlparse.
    new_url = ParseResult(
        parsed_url.scheme, parsed_url.netloc, parsed_url.path,
        parsed_url.params, encoded_get_args, parsed_url.fragment
    ).geturl()

    return new_url

Please be aware that there may be some issues, if you'll find one please let me know and we will make this thing better

4
  • Perhaps add a try except with from urllib.parse to include Python 3 support? Thanks for the snippet, very useful!
    – MattV
    Jul 29, 2015 at 14:01
  • Maybe add imports too ? May 15, 2017 at 11:21
  • 1
    Unencodes encoded urls such as http://stackoverflow.com/with%2Fencoded?data=some&data=values&answe%2rs=false. Also, use three chevrons >>> to help doctests pick up your doctests
    – pelson
    Apr 8, 2020 at 9:00
  • Why not change parsed_get_args = dict(parse_qsl(get_args)) to parsed_get_args = parse_qs(get_args) Apr 23, 2020 at 17:27
61

You want to use URL encoding if the strings can have arbitrary data (for example, characters such as ampersands, slashes, etc. will need to be encoded).

Check out urllib.urlencode:

>>> import urllib
>>> urllib.urlencode({'lang':'en','tag':'python'})
'lang=en&tag=python'

In python3:

from urllib import parse
parse.urlencode({'lang':'en','tag':'python'})
1
29

You can also use the furl module https://github.com/gruns/furl

>>> from furl import furl
>>> print furl('http://example.com/search?q=question').add({'lang':'en','tag':'python'}).url
http://example.com/search?q=question&lang=en&tag=python
0
27

If you are using the requests lib:

import requests
...
params = {'tag': 'python'}
requests.get(url, params=params)
1
  • 1
    @chefhose the question is ... relative to what ? You are not in a web page, there is no context to be relative to. Dec 23, 2019 at 10:05
17

Based on this answer, one-liner for simple cases (Python 3 code):

from urllib.parse import urlparse, urlencode


url = "https://stackoverflow.com/search?q=question"
params = {'lang':'en','tag':'python'}

url += ('&' if urlparse(url).query else '?') + urlencode(params)

or:

url += ('&', '?')[urlparse(url).query == ''] + urlencode(params)
1
  • 4
    I know you mentioned "simple cases", but to clarify: it won't work properly if there is an ? in the anchor (#?stuff). Dec 12, 2016 at 10:57
15

I find this more elegant than the two top answers:

from urllib.parse import urlencode, urlparse, parse_qs

def merge_url_query_params(url: str, additional_params: dict) -> str:
    url_components = urlparse(url)
    original_params = parse_qs(url_components.query)
    # Before Python 3.5 you could update original_params with 
    # additional_params, but here all the variables are immutable.
    merged_params = {**original_params, **additional_params}
    updated_query = urlencode(merged_params, doseq=True)
    # _replace() is how you can create a new NamedTuple with a changed field
    return url_components._replace(query=updated_query).geturl()

assert merge_url_query_params(
    'http://example.com/search?q=question',
    {'lang':'en','tag':'python'},
) == 'http://example.com/search?q=question&lang=en&tag=python'

The most important things I dislike in the top answers (they are nevertheless good):

  • Łukasz: having to remember the index at which the query is in the URL components
  • Sapphire64: the very verbose way of creating the updated ParseResult

What's bad about my response is the magically looking dict merge using unpacking, but I prefer that to updating an already existing dictionary because of my prejudice against mutability.

1
  • I was about to write an answer and found this. (y)
    – Kireeti K
    Sep 10, 2020 at 8:45
11

Yes: use urllib.

From the examples in the documentation:

>>> import urllib
>>> params = urllib.urlencode({'spam': 1, 'eggs': 2, 'bacon': 0})
>>> f = urllib.urlopen("http://www.musi-cal.com/cgi-bin/query?%s" % params)
>>> print f.geturl() # Prints the final URL with parameters.
>>> print f.read() # Prints the contents
5
  • 1
    Can you please give some brief example?
    – z4y4ts
    Mar 24, 2010 at 9:11
  • 1
    f.read() will show you the HTML page. To see the calling url, f.geturl()
    – ccheneson
    Mar 24, 2010 at 9:20
  • 9
    -1 for using a HTTP request for parsing a URL (which is actually basic string manipulation). Plus the actual problem is not considered, because you need to know how the URL looks like to be able to append the query string correctly.
    – poke
    Mar 24, 2010 at 10:11
  • Either the author edited question either this answer is not related to it.
    – simplylizz
    Feb 27, 2013 at 17:20
  • for python 3 this is now: urllib.request.urlopen and urllib.parse.urlencode
    – smoquet
    Oct 21, 2021 at 13:31
10

python3, self explanatory I guess

from urllib.parse import urlparse, urlencode, parse_qsl

url = 'https://www.linkedin.com/jobs/search?keywords=engineer'

parsed = urlparse(url)
current_params = dict(parse_qsl(parsed.query))
new_params = {'location': 'United States'}
merged_params = urlencode({**current_params, **new_params})
parsed = parsed._replace(query=merged_params)

print(parsed.geturl())
# https://www.linkedin.com/jobs/search?keywords=engineer&location=United+States
4
  • Watch out! This approach is using an internal function (indicated by the "_" before the function name) : _replace. It is not recommended to do so, because behavior of these internal functions may change or they may be removed without warning.
    – Adrian
    Nov 5, 2021 at 16:40
  • 2
    According to another stack overflow comment, admittedly from a few years ago, that's not true @GrazingScientist: stackoverflow.com/questions/21628852/… May 7, 2022 at 17:37
  • 1
    @yellow-saint: You are absolutly right, even with Python 3.9 _replace is still a valid public method. I did not know that. So, thanks for pointing this out. Still, this is not the usual case for methods starting with an underscore ("_").
    – Adrian
    May 8, 2022 at 20:11
  • 1
    Just to add, _replace is part of the publicly advertised API, as it's documented in the docs: docs.python.org/3/library/… (One such method is _replace(). The _replace() method will return a new ParseResult object replacing specified fields with new values) Jan 13 at 21:36
7

I liked Łukasz version, but since urllib and urllparse functions are somewhat awkward to use in this case, I think it's more straightforward to do something like this:

params = urllib.urlencode(params)

if urlparse.urlparse(url)[4]:
    print url + '&' + params
else:
    print url + '?' + params
1
  • 5
    How about .query instead of [4] ? Mar 19, 2015 at 17:10
5

Use the various urlparse functions to tear apart the existing URL, urllib.urlencode() on the combined dictionary, then urlparse.urlunparse() to put it all back together again.

Or just take the result of urllib.urlencode() and concatenate it to the URL appropriately.

4

Yet another answer:

def addGetParameters(url, newParams):
    (scheme, netloc, path, params, query, fragment) = urlparse.urlparse(url)
    queryList = urlparse.parse_qsl(query, keep_blank_values=True)
    for key in newParams:
        queryList.append((key, newParams[key]))
    return urlparse.urlunparse((scheme, netloc, path, params, urllib.urlencode(queryList), fragment))
2

In python 2.5

import cgi
import urllib
import urlparse

def add_url_param(url, **params):
    n=3
    parts = list(urlparse.urlsplit(url))
    d = dict(cgi.parse_qsl(parts[n])) # use cgi.parse_qs for list values
    d.update(params)
    parts[n]=urllib.urlencode(d)
    return urlparse.urlunsplit(parts)

url = "http://stackoverflow.com/search?q=question"
add_url_param(url, lang='en') == "http://stackoverflow.com/search?q=question&lang=en"
2

Here is how I implemented it.

import urllib

params = urllib.urlencode({'lang':'en','tag':'python'})
url = ''
if request.GET:
   url = request.url + '&' + params
else:
   url = request.url + '?' + params    

Worked like a charm. However, I would have liked a more cleaner way to implement this.

Another way of implementing the above is put it in a method.

import urllib

def add_url_param(request, **params):
   new_url = ''
   _params = dict(**params)
   _params = urllib.urlencode(_params)

   if _params:
      if request.GET:
         new_url = request.url + '&' + _params
      else:
         new_url = request.url + '?' + _params
   else:
      new_url = request.url

   return new_ur
1
  • urllib has been split up in Python 3. The urllib.urlencode() function is now urllib.parse.urlencode(),
    – Nada Touil
    Apr 6, 2022 at 23:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.