1

I'm trying to package up a script including its dependencies into an executable zip file, following basically the following steps:

$ mkdir build
$ pip install --target build -r requirements.txt
$ cp -r mypackage build
$ echo 'from mypackage import main; main()' > build/__main__.py
$ cd build
$ zip -r myscript-with-dependencies.zip *
$ echo '#!/usr/bin/env python' | cat - myscript-with-dependencies.zip \
    >../dist/myscript

This works, in general, except that .pth files don't seem to be evaled: they're ignored by site.addsitedir(zip_path) (probably because it doesn't seem to have been written to understand zip files).

This is specifically a problem for me because some of my dependencies use setuptools namespace-packages, and its magic happens mostly within .pth files.

Before I go off and write something like addzipsitedir I was wondering if anyone has experience with this kind of problem, am I something obvious? In general the zipimport docs don't say much beyond "it just works!". (Which, again, it mostly does, except for namespace packages and things installed into eggs, etc.)

4
  • You cannot merge the namespaced eggs into one directory structure?
    – Martijn Pieters
    May 5, 2013 at 20:44
  • I can indeed, but I'd like a solution that is more general (and doesn't require different Makefile hacks every time I want to package a new project or add a new dependency.) May 5, 2013 at 20:46
  • That's the work-around I'd think about in any case. :-)
    – Martijn Pieters
    May 5, 2013 at 20:47
  • And it's probably easier than writing addzipsitedir, thank you! May 5, 2013 at 20:50

1 Answer 1

1

Indeed, addsitedir() does not handle zip files; but it should be easy enough to replicate the behavior.

See the site.addsitedir() source code; the code simply attempts to call os.listdir() on the path to then discover .pth files:

try:
    names = os.listdir(sitedir)
except os.error:
    return
dotpth = os.extsep + "pth"
names = [name for name in names if name.endswith(dotpth)]
for name in sorted(names):
    addpackage(sitedir, name, known_paths)

where addpackage() is just site.addpackage(). You can replace the above names listing with a listing of .pth files inside of your zipfile. You'll have to replicate the site.addpackage() behaviour as well, that function expects to be able to read the .pth file.

Simplified, that function does:

with f:
    for n, line in enumerate(f):
        if line.startswith("#"):
            continue
        if line.startswith(("import ", "import\t")):
            exec line
            continue
        line = line.rstrip()
        dir, dircase = makepath(sitedir, line)
        if not dircase in known_paths and os.path.exists(dir):
            sys.path.append(dir)
            known_paths.add(dircase)

with exception handling mixed in. makepath() is site.makepath(), and known_paths ensures that any path found is only added once.

So, in essence, all items named by .pth are added to your sys.path but anything starting with import is executed there and then, giving the .pth file a hook into the site.py loading stage.

The import hook is used by setuptools-based packages to build the namespaced packages; here is one from a package in the zc namespace:

import sys,types,os; p = os.path.join(sys._getframe(1).f_locals['sitedir'], *('zc',)); ie = os.path.exists(os.path.join(p,'__init__.py')); m = not ie and sys.modules.setdefault('zc',types.ModuleType('zc')); mp = (m or []) and m.__dict__.setdefault('__path__',[]); (p not in mp) and mp.append(p)

It creates an empty module object in sys.modules using the sitedir local variable from the addpackage() function. Note the os.path.exists() call; for zipped eggs this will fail (not add the empty module object), so you may need to detect namespaced packages and come up with your own version for these. Any package in the zc namespace just ensures there is a ModuleType() object for the parent namespaces it cares about, with a __path__ attribute pointing to {sitedir}/{packagename}/__init__.py.

The other alternative is to merge the namespaced packages into one directory structure instead.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.