如何创建正确垃圾收集的自定义生成器 class

Question

我正在尝试在 Python 中编写一个作为生成器对象的 class，特别是当它被垃圾收集时 .close() 被调用。这很重要，因为这意味着当生成器被中断时，我可以确保它会自行清理，例如关闭文件或释放锁。

这是一些解释性代码：如果你中断一个生成器，那么当它被垃圾收集时，Python 调用生成器对象上的 .close()，这会向生成器抛出一个 GeneratorExit 错误，可以捕获它以允许清理，比如如下：

from threading import Lock

lock = Lock()

def CustomGenerator(n, lock):
    lock.acquire()
    print("Generator Started: I grabbed a lock")
    try:
        for i in range(n):
            yield i
    except GeneratorExit:
        lock.release()
        print("Generator exited early: I let go of the lock")
        raise
    print("Generator finished successfully: I let go of the lock")

for i in CustomGenerator(100, lock):
    print("Received ", i)
    time.sleep(0.02)
    if i==3:
        break

if not lock.acquire(blocking=False):
    print("Oops: Finished, but lock wasn't released")
else:
    print("Finished: Lock was free")
    lock.release()

Generator Started: I grabbed a lock
Received  0
Received  1
Received  2
Received  3
Generator exited early: I let go of the lock
Finished: Lock was free

但是，如果您尝试通过从 collections.abc.Generator 继承来实现自己的生成器对象，Python 似乎没有注意到它应该在收集对象时调用关闭:

from collections.abc import Generator
class CustomGeneratorClass(Generator):
    def __init__(self, n, lock):
        super().__init__()
        self.lock = lock
        self.lock.acquire()
        print("Generator Class Initialised: I grabbed a lock")
        self.n = n
        self.c = 0

    def send(self, arg):
        value = self.c
        if value >= self.n:
            raise StopIteration
        self.c += 1
        return value

    def throw(self, type, value=None, traceback=None):
        print("Exception Thrown in Generator: I let go of the lock")
        self.lock.release()
        raise StopIteration

for i in CustomGeneratorClass(100, lock):
    print("Received ", i)
    time.sleep(0.02)
    if i==3:
        break

if not lock.acquire(blocking=False):
    print("Oops: Finished, but lock wasn't released")
else:
    print("Finished: Lock was free")
    lock.release()

Generator Class Initialised: I grabbed a lock
Received  0
Received  1
Received  2
Received  3
Oops: Finished, but lock wasn't released

我认为继承 Generator 足以让 python 相信我的 CustomGeneratorClass 是一个生成器并且应该 .close() 在收集垃圾时调用它。

我认为这与以下事实有关：虽然 'generator object' 是某种特殊的 Generator:

from types import GeneratorType

c_gen = CustomGenerator(100)
c_gen_class = CustomGeneratorClass(100)

print("CustomGenerator is a Generator:", isinstance(c_gen, Generator))
print("CustomGenerator is a GeneratorType:",isinstance(c_gen, GeneratorType))

print("CustomGeneratorClass is a Generator:",isinstance(c_gen_class, Generator))
print("CustomGeneratorClass is a GeneratorType:",isinstance(c_gen_class, GeneratorType))

CustomGenerator is a Generator: True
CustomGenerator is a GeneratorType: True
CustomGeneratorClass is a Generator: True
CustomGeneratorClass is a GeneratorType: False

我可以创建一个用户定义的 class 对象 GeneratorType 吗？

关于 python 如何决定在 .close() 上调用什么，我有什么不明白的地方吗？

如何确保在我的自定义生成器上调用 .close()？

此问题与不重复。对于实际制作一个生成器 class，该问题的可接受答案确实推荐了我在这里尝试的结构，它是一个生成器 class 但没有正确地进行垃圾收集，如上面的代码所示.

Answer 1

PEP342，状态：

[generator].__del__() is a wrapper for [generator].close(). This will be called when the generator object is garbage-collected ...

collections.abc 中的生成器 class 没有实现 __del__，它的超classes 或 metaclass 也没有实现。

将 __del__ 的此实现添加到问题中的 class 会导致释放锁：

class CustomGeneratorClass(Generator):

    ...

    def __del__(self):
        self.close()

输出：

Generator Class Initialised: I grabbed a lock
Recieved  0
Recieved  1
Recieved  2
Recieved  3
Exception Thrown in Generator: I let go of the lock
Finished: Lock was free

警告：

我对 Python 中复杂的对象终结没有经验，所以这个建议应该仔细检查，并测试破坏性。特别是，应考虑 language reference 中有关 __del__ 的警告。

更高级别的解决方案是运行上下文管理器中的生成器

with contextlib.closing(CustomGeneratorClass(100, lock)):
    # do stuff

但这很麻烦，并且依赖于代码的用户记得这样做。

如何创建正确垃圾收集的自定义生成器 class

How to create a custom generator class that is correctly garbage collected

python

generator

python-internals