2

I'm studying how python loads modules.

I'm starting from requests that loads urllib3.

I have found those line:

from .packages.six.moves.http_client import (
  IncompleteRead as httplib_IncompleteRead
)

in file called src/urllib3/exceptions.py

That import is a relative one, so I'm looking for the folder src/urllib3/packages/six/moves/http_client.py or src/urllib3/packages/six/moves/http_client/__init__.py

Those files don't exist.

Thankfully src/urllib3/packages/six.py defines a new module loader, so the import .packages.six.moves.http_client requires http_client core python module instead.

But I don't understand how the src/urllib3/packages/six.py is loaded. I've not found that kind of import.

Is there a way to know which module loads src/urllib3/packages/six.py directly?

I have tried with __name__ global variable as suggested here. Its value is urllib3.packages.six but urllib3.packages doesn't load that file.

2 Answers 2

3

The from .packages.six.moves.http_client import ... expression causes .packages.six to be loaded first. Python always loads all packages in a nested package reference to a module.

So .packages.six.moves.http_client causes Python first to look for urllib3.packages, then for urllib3.packages.six, and so on. The import machinery does so by looking for the full name in sys.modules, and if it is not there, triggers a module search and load for each.

The first time this happens, sys.modules['urllib3.packages.six'] doesn't exist yet, the import machinery finds the file urllib3/packages/six.py, imports that, before it'll look for more names.

And, as you discovered, the very act of importing the six.py module file, causes that module to add sys.modules['urllib3.packages.six.moves'] and further references to standard library modules.

Python's import machinery is quite a complex beast; the Python reference documentation covers this comprehensively in The import system; the specific entries to look for are:

A direct call to __import__() performs only the module search and, if found, the module creation operation. While certain side-effects may occur, such as the importing of parent packages, and the updating of various caches (including sys.modules), only the import statement performs a name binding operation.

and under Regular packages

Importing parent.one will implicitly execute parent/__init__.py and parent/one/__init__.py. Subsequent imports of parent.two or parent.three will execute parent/two/__init__.py and parent/three/__init__.py respectively.

and under The module cache:

The first place checked during import search is sys.modules. This mapping serves as a cache of all modules that have been previously imported, including the intermediate paths. So if foo.bar.baz was previously imported, sys.modules will contain entries for foo, foo.bar, and foo.bar.baz. Each key will have as its value the corresponding module object.

(bold emphasis in quoted sections added by me).

Note that everything in the urllib3/packages directory is a vendorized package; a project that normally would be installed independently, but which the urllib3 project has decided to package up with their own distribution to avoid having to worry about what versions to support. six is such an independent project, you can install it from PyPI.

You can find more information on the six.moves virtual package in the six project documentation. It's purpose is to make it easier for library developers to write code that is compatible both with Python 2 and Python 3, without having to worry about what standard library name to import on either version.

0

One notable thing about import function in Python is that it can import not only modules themselves but also variables, classes, functions (generally namespaces) from inside them. In the example, you have provided:

from .packages.six.moves.http_client import (
  IncompleteRead as httplib_IncompleteRead
)

import statement refers to moves variable from src/urllib3/packages/six.py module, which is defined at line 316 of that file and being assigned to an instance of class _MovedItems:

moves = _MovedItems(__name__ + ".moves")

and http_client is a property or method of this class instance

2
  • 1
    The most notable is that the parts before the import in a from ... import ... statement must all be modules, those names will be looked up in sys.modules and no attributes will be consulted. So for this specific situation, where urllib3.packages.six installs virtual modules in sys.modules['urllib3.packages.six.moves'] and onwards, it doesn't matter one bit that you can import attributes from the final module object.
    – Martijn Pieters
    Oct 18, 2018 at 20:38
  • Wow, i didn't know that. You made it Oct 18, 2018 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.