新闻 API - 将输出输出到 Pandas DataFrame

Question

我已成功调用新闻 API 并将结果放入 DataFrame，但仅限于第 1 页。

def get_articles(keyword):

  all_articles = newsapi.get_everything(q=keyword, sources='abc-news-au, news-com-au',
                                      domains='http://www.abc.net.au/news, http://www.news.com.au',
                                      from_param='2018-12-28',
                                      to='2019-01-28',
                                      language='en',
                                      sort_by='popularity',
                                      page=1)

  all_articles = pd.DataFrame(all_articles)
  all_articles = pd.concat([all_articles.drop(['articles'], axis=1), all_articles['articles'].apply(pd.Series)], axis=1)

  return all_articles

它给了我想要的数据框，但是，当我尝试循环浏览以下页面时，我变得不卡住了。

我试过以下方法

empty_list = []

for i in range(1,4,1):
  all_articles = all_articles = newsapi.get_everything(q=keyword, sources='abc-news-au, news-com-au',
                                  domains='http://www.abc.net.au/news, http://www.news.com.au',
                                  from_param='2018-12-28',
                                  to='2019-01-28',
                                  language='en',
                                  sort_by='popularity',
                                  page=i)
  empty_list.append(all_articles)

这returns所有文章，但是它是一个存储在列表中的字典。

[{'articles': [{'author': None,
    'content': 'Updated \r\nJanuary 14, 2019 14:33:00\r\nANZ customers have lost access to banking services at their local post offices after the bank failed to reach an agreement with Australia Post on their Bank@Post service.\r\nThe change, which came into effect last night, wil… [+5084 chars]',
    'description': 'ANZ customers can no longer utilise banking services at their local post offices after the bank failed to reach an agreement with Australia Post on their Bank@Post service.',
    'publishedAt': '2019-01-14T03:14:57Z',
    'source': {'id': 'abc-news-au', 'name': 'ABC News (AU)'},
    'title': "ANZ customers 'furious' as access to Bank@Post cancelled",
    'url': 'https://www.abc.net.au/news/2019-01-14/anz-customers-lose-banking-service-at-australia-post/10713156',
    'urlToImage': 'https://www.abc.net.au/news/image/10710052-16x9-700x394.jpg'},
   {'author': 'Stephen Letts',
    'content': "Posted \r\nJanuary 26, 2019 06:20:15\r\nIf you think AMP's glum market update of an additional 0 million worth of costs to fix its various scandals rules a line under the sordid and sorry mess, think again.\r\nKey points:\r\nRemediation costs for Australia's scand… [+5019 chars]",
    'description': "Australia's six big wealth managers currently have provisions for about .6 billion to fix the scandals that have emerged from the banking royal commission. That could be be woefully inadequate.",
    'publishedAt': '2019-01-25T19:20:15Z',
    'source': {'id': 'abc-news-au', 'name': 'ABC News (AU)'},
    'title': "Wealth managers' remediation costs set to soar",
    'url': 'https://www.abc.net.au/news/2019-01-26/wealth-manager-remediation-costs-set-to-soar/10749810',
    'urlToImage': 'https://www.abc.net.au/news/image/1147126-16x9-700x394.jpg'}]

以前，它只是一本字典[无列表]。

当我做一些转换[类似于上面]时，我得到以下数据帧

问题：

有谁知道更好的方法吗？
如果您要使用当前数据框，您将如何从每一列中提取字典并呈现它以使其看起来像第一个数据框？

如有任何帮助，我们将不胜感激。

PS：如果你想复制，你可以复制我的代码 - 你只需要从 https://newsapi.org/docs/client-libraries/python

获取你自己的 API 密钥

Answer 1

您似乎想提取文章值和 extend 而不是 append:

articles = []

for i in range(1,4,1):
    articles_page = newsapi.get_everything(
            q=keyword,
            sources='abc-news-au, news-com-au',
            domains='http://www.abc.net.au/news, http://www.news.com.au',
            from_param='2018-12-28',
            to='2019-01-28',
            language='en',
            sort_by='popularity',
            page=i)
    articles.extend(articles_page['articles'])

# outside of the loop, create the DataFrame
pd.DataFrame(articles)

新闻 API - 将输出输出到 Pandas DataFrame

News API - Getting output into Pandas DataFrame

python

pandas

dataframe

google-news