bigrquery：使用 geoJSON 文件创建 BigQuery table 不起作用

Question

我想用 geoJSON files 创建一个 BigQuery table，尽管 geoJSON 是 BQ (NEWLINE_DELIMITED_JSON) 和 bq_fields specification, or something coercible to it (like a data frame) bigrquery 包的函数bq_table_create() 不起作用。在我下面的示例中，输出错误是 Erro: Unsupported type: list:

library(sf)
library(bigrquery)
library(DBI)
library(googleAuthR)
library(geojsonsf)
library(geojsonR)


# Convert shapefile to geoJSON 
stands_sel <- st_read(
  "D:/Dropbox/Stinkbug_Ml_detection_CMPC/dashboard/v_08_CMPC/sel_stands_CMPC.shp")

# Open as geoJSON
geo <- sf_geojson(stands_sel)

# Convert geoJSON to data frame 
geo_js_df <- as.data.frame(geojson_wkt(geo))
str(geo_js_df)
# 'data.frame': 2 obs. of  17 variables:
#  $ SISTEMA_PR: chr  "MACRO ESTACA - EUCALIPTO" "SEMENTE - EUCALIPTO"
#  $ ESPECIE   : chr  "SALIGNA" "DUNNI"
#  $ ID_UNIQUE : chr  "BARBANEGRA159A" "CAMPOSECO016A"
#  $ CICLO     : num  2 1
#  $ LOCALIDADE: chr  "BARRA DO RIBEIRO" "DOM FELICIANO"
#  $ ROTACAO   : num  1 1
#  $ CARACTER_1: chr  "Produtivo" "Produtivo"
#  $ VLR_AREA  : num  8.53 28.07
#  $ ID_REGIAO : num  11 11
#  $ CD_USO_SOL: num  2433 9053
#  $ DATA_PLANT: chr  "2008/04/15" "2010/04/15"
#  $ ID_PROJETO: chr  "002" "344"
#  $ CARACTERIS: chr  "Plantio Comercial" "Plantio Comercial"
#  $ PROJETO   : chr  "BARBA NEGRA" "CAMPO SECO"
#  $ ESPACAMENT: chr  "3.00 x 2.50" "3.5 x 2.14"
#  $ CD_TALHAO : chr  "159A" "016A"
#  $ geometry  :List of 2
#   ..$ : 'wkt' chr "MULTIPOLYGON (((-51.2142 -30.3517,-51.2143 -30.3518,-51.2143 -30.3518,-51.2143 -30.3519,-51.2143 -30.3519,-51.2"| __truncated__
#   ..$ : 'wkt' chr "MULTIPOLYGON (((-52.3214 -30.4271,-52.3214 -30.4272,-52.3214 -30.4272,-52.3215 -30.4272,-52.3215 -30.4272,-52.3"| __truncated__
#  - attr(*, "wkt_column")= chr "geometry"

# Insert information inside BQ
bq_conn <-  dbConnect(bigquery(),
                       project = "my-project",
                       use_legacy_sql = FALSE
)

# First create the table
players_table = bq_table(project = "my-project", dataset = "stands_ROI_2021", table = "CF_2021")
bq_table_create(x = players_table, fields = as_bq_fields(geo_js_df))
Erro: Unsupported type: list

Answer 1

您可以使用 bq_table_upload() 语法在 BigQuery 上上传带有列表类型列的数据框。在你的脚本上试试这个而不是 bq_table_create(),

bq_table_upload(players_table, geo_js_df)

为了您的参考，我最后使用带有列表类型列的示例数据进行了尝试：

d <- data.frame(id = 1:2,
                   name = c("Jon", "Mark"),
                   children = I(list(c("Mary", "James"),
                                     c("Greta", "Sally")))
                )

R 控制台：

已创建 BQ table:

编辑：

按照这个 documentation, FeatureCollection is not yet supported in BigQuery, however there is an ongoing feature request you can find here。解决方法是将 GeoJson 文件转换为 BigQuery 换行符分隔 JSON，然后再将其转换为数据帧。

要将 GeoJson 文件转换为 BigQuery 换行符分隔 JSON，请遵循这些 steps：

安装node.js.
添加包：

npm install fs JSONStream line-input-stream yargs

克隆 github 存储库：

git clone https://github.com/mentin/geoscripts.git

cd geoscripts/geojson2bq/

将 GeoJson 文件转换为 BigQuery 换行符分隔 JSON:

node geojson2bqjson.js sel_stands.geojson > out.json

使用新行分隔的 JSON 文件，在 R 控制台中将其转换为数据帧，然后使用 bq_table_upload() 将数据上传到 BigQuery。

library(bigrquery)
library(dplyr)
library(tidyverse)
library(jsonlite)

out <- stream_in(file('out.json'))

projectid<-"my-project"
datasetid<-"my-dataset"

bq_conn <-  dbConnect(bigquery(), 
                      project = projectid,
                      dataset = datasetid, 
                      use_legacy_sql = FALSE)

players_table = bq_table(project = "my-project", dataset = "my-dataset", table = "CF_2021_test5")

bq_table_upload(players_table, out)

bq_table_download(players_table)

R控制台：

BigQuery table:

bigrquery：使用 geoJSON 文件创建 BigQuery table 不起作用

bigrquery: Create a BigQuery table with geoJSON files doesn't work

r

google-bigquery

bigrquery