KeyError: 'users' when using tweepy to scrape the twitter data, how can I bypass such error?

Question

我打算通过 tweepy 包使用学术帐户从 Twitter 上抓取信息，它可以正常工作几个月，但特别是几个月它显示“KeyError：'users'”。我怎样才能绕过这个错误？

这是我执行的代码；

tweets = []

for i,j in zip(start_time,end_time):
    print(i)
    print(j)
    for response in tweepy.Paginator(client.search_all_tweets, 
                                     query = 'ไทรอยด์ -is:retweet lang:th',
                                     user_fields = ['username', 'public_metrics', 'description', 'location'],
                                     tweet_fields = ['created_at', 'geo', 'public_metrics', 'text'],
                                     expansions = ['author_id', 'geo.place_id'],
                                     start_time = i,
                                     end_time = j):
                                     
        time.sleep(1)
        tweets.append(response)
    result = []
    user_dict = {}
    
    for response in tweets:
        
        for user in response.includes['users']:
            user_dict[user.id] = {'username': user.username, 
                                  'followers': user.public_metrics['followers_count'],
                                  'tweets': user.public_metrics['tweet_count'],
                                  'description': user.description,
                                  'location': user.location
                                 }

结果：

2020-08-01T00:00:00Z
2020-08-31T23:59:59Z
Rate limit exceeded. Sleeping for 30 seconds.
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_13308/265973239.py in <module>
     23     for response in tweets:
     24         # Take all of the users, and put them into a dictionary of dictionaries with the info we want to keep
---> 25         for user in response.includes['users']:
     26             user_dict[user.id] = {'username': user.username, 
     27                                   'followers': user.public_metrics['followers_count'],

KeyError: 'users'

Answer 1

您可以让您的代码使用一个空的默认值，这样如果 users 字段不存在，它就不会失败：

for user in response.includes.get('users', ''):
     # do something with user

但是，这只能修复症状。要找出该字段不存在的原因，您可以尝试在使用之前检查密钥（例如 if 'users' in response.includes）并打印出完整的响应数据以获取更多信息。一个提示是您收到的速率限制消息。

KeyError: 'users' when using tweepy to scrape the twitter data, how can I bypass such error?

KeyError: 'users' when using tweepy to scrape the twitter data, how can I bypass such error?

python-3.x

web-scraping

twitter

tweepy