5

Excuse me, what the heck?

>>> import urllib.parse
>>> base = 'http://example.com'
>>> urllib.parse.urljoin(base, 'abc:123')
'http://example.com/abc:123'
>>> urllib.parse.urljoin(base, '123:abc')
'123:abc'
>>> urllib.parse.urljoin(base + '/', './123:abc')
'http://example.com/123:abc'

python3.7 documentation says:

Changed in version 3.5: Behaviour updated to match the semantics defined in RFC 3986.

Which part of that RFC enforces such madness, and whether it should be considered a bug?

5
  • 1
    That does indeed look like a bug to me. RFC 3986 states "Scheme names consist of a sequence of characters beginning with a letter", which 123 clearly does not and thus isn't a valid scheme. So, as far as I can tell, urllib.parse.urljoin(base, '123:abc') should actually return 'http://example.com/123:abc' and urllib.parse.urljoin(base, 'abc:123') should return 'abc:123'. But then again, while that's a valid URI, it's not a valid URL...
    – Aran-Fey
    Commented Mar 17, 2019 at 0:52
  • @gdlmx specifically, Python 3.7.2
    – ratijas
    Commented Mar 17, 2019 at 1:20
  • getting the same with 2.7.15 as well
    – ratijas
    Commented Mar 17, 2019 at 1:22
  • I have tested in 3.7.2 and got this behavior. But the last line in your post is wrong: 'http://example.com/path/123:abc'.
    – gdlmx
    Commented Mar 17, 2019 at 1:43
  • @gdlmx fixed the last line. Indeed, I've played with it for a while and basically forgot to strip the example.
    – ratijas
    Commented Mar 17, 2019 at 2:47

1 Answer 1

4

Which part of that RFC enforces such madness?

This behavior is correct and consistent with other implementations, as indicated by RFC3986:

A path segment that contains a colon character (e.g., "this:that") cannot be used as the first segment of a relative-path reference, as it would be mistaken for a scheme name. Such a segment must be preceded by a dot-segment (e.g., "./this:that") to make a relative-path reference.

It's been already discussed in another post:

Colons are allowed in the URI path. But you need to be careful when writing relative URI paths with a colon since it is not allowed when used like this:

<a href="tag:sample">

In this case tag would be interpreted as the URI’s scheme. Instead you need to write it like this:

<a href="./tag:sample">

Usage of urljoin

The function urljoin simply treats both arguments as URL (without any presumption). It requires that their schemes to be identical or the second one to represent a relative URI path. Otherwise, it only returns the second argument (although, IMHO, it should raise an error). You can better understand the logic by looking into the source of urljoin.

def urljoin(base, url, allow_fragments=True):
    """Join a base URL and a possibly relative URL to form an absolute
    interpretation of the latter."""
    ...
    bscheme, bnetloc, bpath, bparams, bquery, bfragment = \
            urlparse(base, '', allow_fragments)
    scheme, netloc, path, params, query, fragment = \
            urlparse(url, bscheme, allow_fragments)

    if scheme != bscheme or scheme not in uses_relative:
        return _coerce_result(url)

The results of the parser routine urlparse are as follow:

>>> from urllib.parse import urlparse
>>> urlparse('123:abc')
ParseResult(scheme='123', netloc='', path='abc', params='', query='', fragment='')
>>> urlparse('abc:123')
ParseResult(scheme='', netloc='', path='abc:123', params='', query='', fragment='')
>>> urlparse('abc:a123')
ParseResult(scheme='abc', netloc='', path='a123', params='', query='', fragment='')
2
  • Sooo... basically what you are saying is: it's rather a problem with urlparse, and rather an unfixable one because it deals with sneaky edge cases.
    – ratijas
    Commented Mar 17, 2019 at 2:51
  • 1
    No. I don't think it's a problem of either urlparse or urljoin. It's a problem of your usage of them. Please use urllib.parse.quote to escape colon in your path!
    – gdlmx
    Commented Mar 20, 2019 at 17:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.