sentry-youtrack plugin: PicklingError: Can't pickle <type 'generator'>: attribute lookup __builtin__.generator failed

Question

• 哨兵：8.4.0
• sentry-youtrack 插件：0.3.1
• youtrack-6.5.17105
• python-内存缓存：1.53

我正在尝试集成 youtrack into sentry using this 插件。

问题是当我们点击更多 --> 创建 YouTrack 问题时页面似乎挂起。查看系统日志，我看到了这个：

Traceback (most recent call last):
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/core/handlers/base.py", line 112, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/views/generic/base.py", line 69, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/utils/decorators.py", line 29, in _wrapper
    return bound_func(*args, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/utils/decorators.py", line 99, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/utils/decorators.py", line 25, in bound_func
    return func(self, *args2, **kwargs2)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/web/frontend/base.py", line 172, in dispatch
    return self.handle(request, *args, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/web/frontend/group_plugin_action.py", line 25, in handle
    response = plugin.get_view_response(request, group)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/plugin.py", line 113, in get_view_response
    return super(YouTrackPlugin, self).get_view_response(request, group)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/plugins/base/v1.py", line 296, in get_view_response
    response = self.view(request, group)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/plugin.py", line 131, in view
    return view(request, group, **kwargs)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/plugins/bases/issue.py", line 169, in view
    form = self.get_new_issue_form(request, group, event)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/plugin.py", line 77, in get_new_issue_form
    project_fields=self.get_project_fields(group.project),
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/plugin.py", line 57, in get_project_fields
    return cached_fields(self.get_option('ignore_fields', project))
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/utils.py", line 16, in wrapper
    cache.set(key, result, timeout)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/core/cache/backends/memcached.py", line 82, in set
    self._cache.set(key, value, self._get_memcache_timeout(timeout))
  File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 740, in set
    return self._set("set", key, val, time, min_compress_len, noreply)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1060, in _set
    return _unsafe_set()
  File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set
    store_info = self._val_to_store_info(val, min_compress_len)
  File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info
    pickler.dump(val)
PicklingError: Can't pickle <type 'generator'>: attribute lookup __builtin__.generator failed

追溯(https://github.com/bogdal/sentry-youtrack/blob/master/sentry_youtrack/utils.py#L6):

def cache_this(timeout=60):
    def decorator(func):
        def wrapper(*args, **kwargs):
            def get_cache_key(*args, **kwargs):
                params = list(args) + kwargs.values()
                return md5("".join(map(str, params))).hexdigest()
            key = get_cache_key(func.__name__, *args, **kwargs)
            result = cache.get(key)
            if not result:
                result = func(*args, **kwargs)
                cache.set(key, result, timeout)
            return result
        return wrapper
    return decorator

我知道我们收到这个错误是因为 result 是一个生成器。

https://github.com/bogdal/sentry-youtrack/blob/master/sentry_youtrack/plugin.py#L51:

def get_project_fields(self, project):
    @cache_this(600)
    def cached_fields(ignore_fields):
        yt_client = self.get_youtrack_client(project)
        return yt_client.get_project_fields(
            self.get_option('project', project), ignore_fields)
    return cached_fields(self.get_option('ignore_fields', project))

https://github.com/bogdal/sentry-youtrack/blob/master/sentry_youtrack/youtrack.py#L198:

def get_project_fields(self, project_id, ignore_fields=None):
    ignore_fields = ignore_fields or []
    for field in self.get_project_fields_list(project_id):
        if not field['name'] in ignore_fields:
            yield self._get_custom_project_field_details(field)

所以，我正在尝试将其转换为列表：

        if not result:
            result = func(*args, **kwargs)
            cache.set(key, [f for f in result], timeout)
        return result

但仍然出现同样的错误：

      File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/utils.py", line 16, in wrapper
        cache.set(key, [f for f in result], timeout)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/core/cache/backends/memcached.py", line 82, in set
        self._cache.set(key, value, self._get_memcache_timeout(timeout))
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 740, in set
        return self._set("set", key, val, time, min_compress_len, noreply)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1060, in _set
        return _unsafe_set()
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set
        store_info = self._val_to_store_info(val, min_compress_len)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info
        pickler.dump(val)
    PicklingError: Can't pickle <type 'generator'>: attribute lookup __builtin__.generator failed

然后我尝试将 val 的值记录到一个文件中：

        try:
            pickler.dump(val)
        except Exception:
            with open('/tmp/quanta.log', 'a+') as f:
                f.write(str(val))

但是没有创建该文件。奇怪的是回溯说错误发生在与之前相同的行：

      File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../sentry_youtrack/utils.py", line 16, in wrapper
        cache.set(key, [f for f in result], timeout)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/sentry/../django/core/cache/backends/memcached.py", line 82, in set
        self._cache.set(key, value, self._get_memcache_timeout(timeout))
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 740, in set
        return self._set("set", key, val, time, min_compress_len, noreply)
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1060, in _set
        msg = msg[1]
      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 1034, in _unsafe_set

      File "/usr/local/sentry/local/lib/python2.7/site-packages/memcache.py", line 998, in _val_to_store_info
        try:
    PicklingError: Can't pickle <type 'generator'>: attribute lookup __builtin__.generator failed

那么，我有两个问题：

1. 为什么转换成列表后还是报错Can't pickle <type 'generator'>...？
2. 如何在调用 pickler.dump(val) 之前调试这种情况以了解 val 的值？

Answer 1

这是 sentry-youtrack 的错误，它不应该缓存生成器对象。 python-memcached 可能有一个 pyc 文件，这就是为什么它没有像您修改的那样转储值。你添加了 (i for i in list) 这也是一个生成器。

您应该使用 getsentry/sentry-youtrack，因为它具有用于缓存生成器的 correct fix。