gevent monkey 是否修补了 Queue.put yield（上下文切换）？

Question

import gevent.monkey
gevent.monkey.patch_all()
import gevent
from queue import Queue
import random
import time


def getter(q):
    while True:
        print('getting')
        v = q.get()
        print(f'got {v}')

def putter(q):
    while True:
        print(f'start putting')
        v = int(random.random() * 1000)
        # `put_nowait` also seems to yield
        # q.put(v)
        q.put_nowait(v)
        print(f'done putting with {v}')

        if random.random() < 0.5:
            print(f'yield')
            time.sleep(0)


queue = Queue()


gevent.spawn(getter, queue)
gevent.spawn(putter, queue)

time.sleep(1000)

不管我用queue.put还是queue.put_nowait，我看到的日志都是

# start putting
# got 25
# getting
# done putting with 535

这是否表明 gevent 可能会在每次执行时进行上下文切换queue.put？

更新

我稍微修改了一下代码

flag = True

def getter(q):
    while True:
        print('getting')
        global flag
        flag = False
        v = q.get()
        print(f'got {v}')

def putter(q):
    v = 0
    while True:
        print(f'start putting {v}')
        global flag
        flag = True
        # `put_nowait` also seems to yield
        # q.put(v)
        q.put_nowait(v)
        if not flag:
            raise Exception('yield happened')

        print(f'done putting with {v}')
        v += 1

        time.sleep(0)
        # If I sleep with non-zero, the above seems to only yield once at the beginning.
        # time.sleep(0.000001)


queue = Queue()


def myTracer(event, args):
    src, target = args
    if event == "switch":
        # print("from %s switch to %s" % (src, target))
        # Print to stdout like the rest of the code. Otherwise the order of stdout & stderr is not guaranteed.
        traceback.print_stack(file=sys.stdout)
    elif event == "throw":
        print("from %s throw exception to %s" % (src, target))


# greenlet.settrace(myTracer)

gevent.spawn(getter, queue)
putter(queue)

已修补 Python 队列，

如果我 sleep(0)，它可能会进行上下文切换。
如果我sleep(0.0000001)，它只会在开始时产生一次。
如果我根本不睡觉，它只会在开始时屈服一次，getter之后就没有机会运行了。

查看堆栈跟踪，我发现 Queue.put 调用了 notify，后者调用了 lock.acquire(0)。然后对其进行了修补，以便在 gevent/thread.py

中调用 sleep()

如果我使用 gevent.queue.Queue 而不是 Python Queue，它似乎不会进行上下文切换。

Answer 1

您可以使用 greenlet.settrace 和回调函数来检测上下文切换。
将其添加到您的代码中表明 put 和 put_nowait 都进行上下文切换。

...

def myTracer(event, args):
    src, target = args
    if event == "switch":
        print("from %s switch to %s" % (src, target))
    elif event == "throw":
        print("from %s throw exception to %s" % (src, target))


greenlet.settrace(myTracer)


queue = Queue()
gevent.spawn(getter, queue)
gevent.spawn(putter, queue)

time.sleep(5)

你会在标准输出中看到很多 'switching' 调试消息：

...
done putting with 106
start putting
from <Greenlet at 0x10418e150: putter(<queue.Queue object at 0x1034d9460>)> switch to <Hub '' at 0x10416a400 select default pending=0 ref=3 thread_ident=0x106d15dc0>
from <Hub '' at 0x10416a400 select default pending=0 ref=1 thread_ident=0x106d15dc0> switch to <Greenlet at 0x10418e150: putter(<queue.Queue object at 0x1034d9460>)>
done putting with 487
start putting
....

Edit/Note:

我正在使用 Python3.8 和 gevent==20.9.0。测试时，我删除了整个 if-condition if random.random() ...，但将 stdout 管道传输到文件并搜索 getter 上下文切换和 getting 两者都存在于 stdout.

我没有进一步调查，但如果你看一下 source code 你会发现有一个显式调用 getter.switch(getter) 可能这就是导致上下文切换的原因。

gevent monkey 是否修补了 Queue.put yield（上下文切换）？

Does gevent monkey patched Queue.put yield (context switch)?

python

queue

multithreading

monkeypatching

gevent

更新