使用 R 中 JSON 文件中存储的文本创建语料库

Question

我有几个 JSON 文件，其中的文本分为 date、body 和 title。例如考虑：

{"date": "December 31, 1990, Monday, Late Edition - Final", "body": "World stock markets begin 1991 facing the threat of a war in the Persian Gulf, recessions or economic slowdowns around the world, and dismal earnings -- the same factors that drove stock markets down sharply in 1990.  Finally, there is the problem of the Soviet Union, the wild card in everyone's analysis. It is a country whose problems could send stock markets around the world reeling if something went seriously awry. With Russia about to implode, that just adds to the risk premium, said Mr. Dhar. LOAD-DATE: December 30, 1990 ", "title": "World Markets;"}
{"date": "December 30, 1992, Sunday, Late Edition - Final", "body": "DATELINE: CHICAGO Gleaming new tractors are becoming more familiar sights on America's farms. Sales and profits at the three leading United States tractor makers -- Deere & Company, the J.I. Case division of Tenneco Inc. and the Ford Motor Company's Ford New Holland division -- are all up, reflecting renewed agricultural prosperity after the near-depression of the early and mid-1980's. But the recovery in the tractor business, now in its third year, is fragile.  Tractor makers hope to install computers that can digest this information, then automatically concentrate the application of costly fertilizer and chemicals on the most productive land. Within the next 15 years, that capability will be commonplace, predicted Mr. Ball. LOAD-DATE: December 30, 1990 ", "title": "All About/Tractors;"}

我有 3 份不同的报纸，它们的文件分别包含 1989 年至 2016 年期间产生的所有文本。我的最终目标是将所有文本组合成一个语料库。我已经在 Python 中使用 pandas 库完成了它，我想知道它是否可以在 R 中类似地完成。这是我在 R 中使用循环的代码：

for (i in 1989:2016){
  df0 = pd.DataFrame([json.loads(l) for l in open('NYT_%d.json' % i)])
  df1 = pd.DataFrame([json.loads(l) for l in open('USAT_%d.json' % i)])
  df2 = pd.DataFrame([json.loads(l) for l in open('WP_%d.json' % i)])
  appended_data.append(df0)
  appended_data.append(df1)
  appended_data.append(df2)
}

Answer 1

R 中有很多选项可以读取 json 文件并将其转换为数据。frame/data.table.

这里使用 jsonlite 和 data.table:

library(data.table)
library(jsonlite)
res <- lapply(1989:2016,function(i){
  ff <- c('NYT_%d.json','USAT_%d.json' ,'WP_%d.json')
  list_files_paths <- sprintf(ff,i)
  rbindlist(lapply(list_files_paths,fromJSON))
  })

这里res是data.table的列表。如果您想将所有 data.table 聚合到一个 data.table 中：

  rbindlist(res)

Answer 2

使用 jsonlite::stream_in 读取您的文件并使用 jsonlite::rbind.pages 合并它们。

Answer 3

使用 ndjson::stream_in 比 jsonlite::stream_in 阅读起来更快更流畅:-)

使用 R 中 JSON 文件中存储的文本创建语料库

Creating a corpus out of texts stored in JSON files in R

json

r

corpus