5

I'm wondering about code like this:

def all_lines(filename):
    with open(filename) as infile:
        yield from infile

The point of a context manager is to have explicit control over the lifetime of some form of state, e.g. a file handle. A generator, on the other hand, keeps its state until it is exhausted or deleted.

I do know that both cases work in practice. But I'm worried about whether it is a good idea. Consider for example this:

def all_first_lines(filenames):
    return [next(all_lines(filename), None) for filename in filenames]

I never exhaust the generators. Instead, their state is destroyed when the generator object is deleted. This works fine in reference-counted implementations like CPython, but what about garbage-collected implementations? I'm practically relying on the reference counter for managing state, something that context managers were explicitly designed to avoid!

And even in CPython it shouldn't be too hard to construct cases were a generator is part of a reference cycle and needs the garbage collector to be destroyed.

To summarize: Would you consider it prudent to avoid context managers in generators, for example by refactoring the above code into something like this?

def all_lines(filename):
    with open(filename) as infile:
        return infile.readlines()

def first_line(filename):
    with open(filename) as infile:
        return next(infile, None)

def all_first_lines(filenames):
    return [first_line(filename) for filename in filenames]
1
  • Not so much "explicit" as "predictable". The file is guaranteed to be closed after the with statement completes, which will be whenever the generator created by all_lines is finalized, whenever that may be.
    – chepner
    Jan 17, 2022 at 17:20

2 Answers 2

1

While it does indeed extend the lifetime of the object until the generator exits or is destroyed, it also can make the generators clearer to work with.

Consider creating the generators under an outer with and passing the file as an argument instead of them opening it. Now the file is invalid for use after the context manager is exited, even though the generators can still be seen as usable.

If limiting the time for how long the handles are held is important, you can explicitly close the generators using the close method after you are done with them.

This is a similar problem to what trio tries to solve with its nurseries for asynchronous tasks, where the nursery context manager waits for every task spawned from that nursery to exit before proceeding, the tutorial example illustrates this. This blog post by the author can provide some reasoning for the way it's done in trio which can be an interesting read that's somewhat related to the problem.

1

There are two answers to your question :

  • the absolutist : indeed, the context managers will not serve their role, the GC will have to clean the mess that should not have happened
  • the pragmatic : true, but is it actually a problem ? Your file handle will get released a few milliseconds later, what's the bother ? Does it have a measurable impact on production, or is it just bikeshedding ?

I'm not an expert to Python alt implementations' differences (see this page for PyPy's example), but I posit that this lifetime problem will not occur in 99% of cases. If you happen to hit in prod, then yes, you should address it (either with your proposal, or a mix of generator with context manager) otherwise, why bother ? I mean it in a kind way : your point is strictly valid, but irrelevant to most cases.

2
  • Welp, there are certainly worse context managers to lose track of than file. Mutex locks come to mind. But I see your point.
    – Homer512
    Jan 17, 2022 at 16:44
  • Yes, but the same thing goes for mutexes : indeed they may be released later then sooner, so it may hurt performance when doing heavy/frequent threading, but you already bought Python's overhead, waiting (in the worst case) for the GC to happen does not looks horrible in comparison. Ss it is explained in the PyPy link, there are cases where it could bite you back, and you should fix around the language. This, to me, is an instance of "worse is better" (the technical solution is imperfect, but it works).
    – Lenormju
    Jan 17, 2022 at 16:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.