如何获取单个 table 的 BigQuery 存储大小
How to get BigQuery storage size for a single table
我想计算 table 大查询存储的 Google 明智成本，但我不知道如何单独查看每个 table 的存储大小。
有几种方法可以做到这一点，但请注意 table 字节的大小 属性 对于主动接收流式插入的 table 是不可用的。
一个。使用BQ command line tool, and the JQ linux库解析JSON.
bq --format=json show publicdata:samples.gsod | jq '.numBytes | tonumber'
这个输出：
17290009238
乙。使用 REST api 进行 Tables:get 调用
GET https://www.googleapis.com/bigquery/v2/projects/projectId/datasets/datasetId/tables/tableId
这 return 是一个完整的 JSON，您可以解析并获得 numBytes。
{
"kind": "bigquery#table",
"description": "This dataset contains weather information collected by NOAA, such a…",
"creationTime": "1335916040125",
"tableReference": {
"projectId": "publicdata",
"tableId": "gsod",
"datasetId": "samples"
},
"numRows": "114420316",
"numBytes": "17290009238",
"etag": "\"Gn3Hpo5WaKnpFuT457VBDNMgZBw/MTQxMzkzNzk4Nzg0Ng\"",
"location": "US",
"lastModifiedTime": "1413937987846",
"type": "TABLE",
"id": "publicdata:samples.gsod",
"selfLink": "https://www.googleapis.com/bigquery/v2/projects/publicdata/datasets…",
"schema": {
"fields": [
{
"description": "The World Meteorological Organization (WMO) / DATSAV3 station numbe…",
"type": "INTEGER",
"name": "station_number",
"mode": "REQUIRED"
},
{
"description": "The Weather-Bureau-Army-Navy (WBAN) station number where the data w…",
"type": "INTEGER",
"name": "wban_number",
"mode": "NULLABLE"
},
{
"description": "The year the data was collected in",
"type": "INTEGER",
"name": "year",
"mode": "REQUIRED"
},
{
"description": "The month the data was collected in",
"type": "INTEGER",
"name": "month",
"mode": "REQUIRED"
},
{
"description": "The day the data was collected in.",
"type": "INTEGER",
"name": "day",
"mode": "REQUIRED"
},
{
"description": "The mean temperature of the day in degrees Fahrenheit, accurate to …",
"type": "FLOAT",
"name": "mean_temp",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_temp.",
"type": "INTEGER",
"name": "num_mean_temp_samples",
"mode": "NULLABLE"
},
{
"description": "The mean dew point of the day in degrees Fahrenheit, accurate to on…",
"type": "FLOAT",
"name": "mean_dew_point",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_dew_point.",
"type": "INTEGER",
"name": "num_mean_dew_point_samples",
"mode": "NULLABLE"
},
{
"description": "The mean sea level pressure of the day in millibars, accurate to on…",
"type": "FLOAT",
"name": "mean_sealevel_pressure",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_sealevel_pressure…",
"type": "INTEGER",
"name": "num_mean_sealevel_pressure_samples",
"mode": "NULLABLE"
},
{
"description": "The mean station pressure of the day in millibars, accurate to one …",
"type": "FLOAT",
"name": "mean_station_pressure",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_station_pressure.…",
"type": "INTEGER",
"name": "num_mean_station_pressure_samples",
"mode": "NULLABLE"
},
{
"description": "The mean visibility of the day in miles, accurate to one tenth of a…",
"type": "FLOAT",
"name": "mean_visibility",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_visibility.",
"type": "INTEGER",
"name": "num_mean_visibility_samples",
"mode": "NULLABLE"
},
{
"description": "The mean wind speed of the day in knots, accurate to one tenth of a…",
"type": "FLOAT",
"name": "mean_wind_speed",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_wind_speed.",
"type": "INTEGER",
"name": "num_mean_wind_speed_samples",
"mode": "NULLABLE"
},
{
"description": "The maximum sustained wind speed reported on the day in knots, accu…",
"type": "FLOAT",
"name": "max_sustained_wind_speed",
"mode": "NULLABLE"
},
{
"description": "The maximum wind gust speed reported on the day in knots, accurate …",
"type": "FLOAT",
"name": "max_gust_wind_speed",
"mode": "NULLABLE"
},
{
"description": "The maximum temperature of the day in degrees Fahrenheit, accurate …",
"type": "FLOAT",
"name": "max_temperature",
"mode": "NULLABLE"
},
{
"description": "Indicates the source of max_temperature.",
"type": "BOOLEAN",
"name": "max_temperature_explicit",
"mode": "NULLABLE"
},
{
"description": "The minimum temperature of the day in degrees Fahrenheit, accurate …",
"type": "FLOAT",
"name": "min_temperature",
"mode": "NULLABLE"
},
{
"description": "Indicates the source of min_temperature.",
"type": "BOOLEAN",
"name": "min_temperature_explicit",
"mode": "NULLABLE"
},
{
"description": "The total precipitation of the day in inches, accurate to one hundr…",
"type": "FLOAT",
"name": "total_precipitation",
"mode": "NULLABLE"
},
{
"description": "The snow depth of the day in inches, accurate to one tenth of an in…",
"type": "FLOAT",
"name": "snow_depth",
"mode": "NULLABLE"
},
{
"description": "Indicates if fog was reported on this day.",
"type": "BOOLEAN",
"name": "fog",
"mode": "NULLABLE"
},
{
"description": "Indicates if rain was reported on this day.",
"type": "BOOLEAN",
"name": "rain",
"mode": "NULLABLE"
},
{
"description": "Indicates if snow was reported on this day.",
"type": "BOOLEAN",
"name": "snow",
"mode": "NULLABLE"
},
{
"description": "Indicates if hail was reported on this day.",
"type": "BOOLEAN",
"name": "hail",
"mode": "NULLABLE"
},
{
"description": "Indicates if thunder was reported on this day.",
"type": "BOOLEAN",
"name": "thunder",
"mode": "NULLABLE"
},
{
"description": "Indicates if a tornado was reported on this day.",
"type": "BOOLEAN",
"name": "tornado",
"mode": "NULLABLE"
}
]
}
}
摄氏度。有元 table 称为 __TABLES__ 和 __TABLES_SUMMARY__
您可以运行这样的查询：
SELECT size_bytes FROM <dataset>.__TABLES__ WHERE table_id='mytablename'
该查询的 __TABLES__ 部分可能看起来很陌生。 __TABLES_SUMMARY__ 是一个元 table，包含数据集中有关 table 的信息。你可以自己使用这个 meta-table。例如，查询 SELECT * FROM publicdata:samples.__TABLES_SUMMARY__ 将 return 关于 publicdata:samples 数据集中 table 的元数据。你也可以做 SELECT * FROM publicdata:samples.__TABLES__
可用字段：
__TABLES_SUMMARY__ meta-table 的字段（在 TABLE_QUERY 查询中都可用）包括：
table_id: table.creation_time：时间，自 1970 年 1 月 1 日以来的毫秒数，创建了 table。这与 table.creation_time 字段相同
type: 是view(2)还是regular table(1).
以下字段 在 TABLE_QUERY() 中不可用，因为它们是 __TABLES__ 但不是 __TABLES_SUMMARY__ 的成员。它们被保留在这里是为了历史利益和部分记录 __TABLES__ metatable:
last_modified_time：时间，自 1970 年 1 月 1 日以来的毫秒数，table 已更新（元数据或 table 内容）。请注意，如果您使用 tabledata.insertAll() 将记录流式传输到您的 table，这可能会过时几分钟。row_count：table.size_bytes：table.
您可以使用命令行工具执行此操作
bq show ds_name.table_name
它将显示有关 table 的一些信息，包括 "Total Bytes"。参考这里https://cloud.google.com/bigquery/bq-command-line-tool
或者从 GUI 中，您可以使用内部元数据 table __TABLES__ ，例如，这将为您提供以 GB 为单位的大小：
select
sum(size_bytes)/pow(10,9) as size
from
<your_dataset>.__TABLES__
where
table_id = '<your_table>'
只需使用命令行工具即可。您将看到 table，其中包含 Last-Modified、Schema、Total Rows、Total Bytes 等列。
- 单身Table
bq show dataset_name.table_name
- 所有 Tables
bq show dataset_name.*
我想计算 table 大查询存储的 Google 明智成本，但我不知道如何单独查看每个 table 的存储大小。
有几种方法可以做到这一点，但请注意 table 字节的大小 属性 对于主动接收流式插入的 table 是不可用的。
一个。使用BQ command line tool, and the JQ linux库解析JSON.
bq --format=json show publicdata:samples.gsod | jq '.numBytes | tonumber'
这个输出：
17290009238
乙。使用 REST api 进行 Tables:get 调用
GET https://www.googleapis.com/bigquery/v2/projects/projectId/datasets/datasetId/tables/tableId
这 return 是一个完整的 JSON，您可以解析并获得 numBytes。
{
"kind": "bigquery#table",
"description": "This dataset contains weather information collected by NOAA, such a…",
"creationTime": "1335916040125",
"tableReference": {
"projectId": "publicdata",
"tableId": "gsod",
"datasetId": "samples"
},
"numRows": "114420316",
"numBytes": "17290009238",
"etag": "\"Gn3Hpo5WaKnpFuT457VBDNMgZBw/MTQxMzkzNzk4Nzg0Ng\"",
"location": "US",
"lastModifiedTime": "1413937987846",
"type": "TABLE",
"id": "publicdata:samples.gsod",
"selfLink": "https://www.googleapis.com/bigquery/v2/projects/publicdata/datasets…",
"schema": {
"fields": [
{
"description": "The World Meteorological Organization (WMO) / DATSAV3 station numbe…",
"type": "INTEGER",
"name": "station_number",
"mode": "REQUIRED"
},
{
"description": "The Weather-Bureau-Army-Navy (WBAN) station number where the data w…",
"type": "INTEGER",
"name": "wban_number",
"mode": "NULLABLE"
},
{
"description": "The year the data was collected in",
"type": "INTEGER",
"name": "year",
"mode": "REQUIRED"
},
{
"description": "The month the data was collected in",
"type": "INTEGER",
"name": "month",
"mode": "REQUIRED"
},
{
"description": "The day the data was collected in.",
"type": "INTEGER",
"name": "day",
"mode": "REQUIRED"
},
{
"description": "The mean temperature of the day in degrees Fahrenheit, accurate to …",
"type": "FLOAT",
"name": "mean_temp",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_temp.",
"type": "INTEGER",
"name": "num_mean_temp_samples",
"mode": "NULLABLE"
},
{
"description": "The mean dew point of the day in degrees Fahrenheit, accurate to on…",
"type": "FLOAT",
"name": "mean_dew_point",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_dew_point.",
"type": "INTEGER",
"name": "num_mean_dew_point_samples",
"mode": "NULLABLE"
},
{
"description": "The mean sea level pressure of the day in millibars, accurate to on…",
"type": "FLOAT",
"name": "mean_sealevel_pressure",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_sealevel_pressure…",
"type": "INTEGER",
"name": "num_mean_sealevel_pressure_samples",
"mode": "NULLABLE"
},
{
"description": "The mean station pressure of the day in millibars, accurate to one …",
"type": "FLOAT",
"name": "mean_station_pressure",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_station_pressure.…",
"type": "INTEGER",
"name": "num_mean_station_pressure_samples",
"mode": "NULLABLE"
},
{
"description": "The mean visibility of the day in miles, accurate to one tenth of a…",
"type": "FLOAT",
"name": "mean_visibility",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_visibility.",
"type": "INTEGER",
"name": "num_mean_visibility_samples",
"mode": "NULLABLE"
},
{
"description": "The mean wind speed of the day in knots, accurate to one tenth of a…",
"type": "FLOAT",
"name": "mean_wind_speed",
"mode": "NULLABLE"
},
{
"description": "The number of observations used to calculate mean_wind_speed.",
"type": "INTEGER",
"name": "num_mean_wind_speed_samples",
"mode": "NULLABLE"
},
{
"description": "The maximum sustained wind speed reported on the day in knots, accu…",
"type": "FLOAT",
"name": "max_sustained_wind_speed",
"mode": "NULLABLE"
},
{
"description": "The maximum wind gust speed reported on the day in knots, accurate …",
"type": "FLOAT",
"name": "max_gust_wind_speed",
"mode": "NULLABLE"
},
{
"description": "The maximum temperature of the day in degrees Fahrenheit, accurate …",
"type": "FLOAT",
"name": "max_temperature",
"mode": "NULLABLE"
},
{
"description": "Indicates the source of max_temperature.",
"type": "BOOLEAN",
"name": "max_temperature_explicit",
"mode": "NULLABLE"
},
{
"description": "The minimum temperature of the day in degrees Fahrenheit, accurate …",
"type": "FLOAT",
"name": "min_temperature",
"mode": "NULLABLE"
},
{
"description": "Indicates the source of min_temperature.",
"type": "BOOLEAN",
"name": "min_temperature_explicit",
"mode": "NULLABLE"
},
{
"description": "The total precipitation of the day in inches, accurate to one hundr…",
"type": "FLOAT",
"name": "total_precipitation",
"mode": "NULLABLE"
},
{
"description": "The snow depth of the day in inches, accurate to one tenth of an in…",
"type": "FLOAT",
"name": "snow_depth",
"mode": "NULLABLE"
},
{
"description": "Indicates if fog was reported on this day.",
"type": "BOOLEAN",
"name": "fog",
"mode": "NULLABLE"
},
{
"description": "Indicates if rain was reported on this day.",
"type": "BOOLEAN",
"name": "rain",
"mode": "NULLABLE"
},
{
"description": "Indicates if snow was reported on this day.",
"type": "BOOLEAN",
"name": "snow",
"mode": "NULLABLE"
},
{
"description": "Indicates if hail was reported on this day.",
"type": "BOOLEAN",
"name": "hail",
"mode": "NULLABLE"
},
{
"description": "Indicates if thunder was reported on this day.",
"type": "BOOLEAN",
"name": "thunder",
"mode": "NULLABLE"
},
{
"description": "Indicates if a tornado was reported on this day.",
"type": "BOOLEAN",
"name": "tornado",
"mode": "NULLABLE"
}
]
}
}
摄氏度。有元 table 称为 __TABLES__ 和 __TABLES_SUMMARY__
您可以运行这样的查询：
SELECT size_bytes FROM <dataset>.__TABLES__ WHERE table_id='mytablename'
该查询的 __TABLES__ 部分可能看起来很陌生。 __TABLES_SUMMARY__ 是一个元 table，包含数据集中有关 table 的信息。你可以自己使用这个 meta-table。例如，查询 SELECT * FROM publicdata:samples.__TABLES_SUMMARY__ 将 return 关于 publicdata:samples 数据集中 table 的元数据。你也可以做 SELECT * FROM publicdata:samples.__TABLES__
可用字段：
__TABLES_SUMMARY__ meta-table 的字段（在 TABLE_QUERY 查询中都可用）包括：
table_id: table. 的名称
creation_time：时间，自 1970 年 1 月 1 日以来的毫秒数，创建了 table。这与 table. 上的 type: 是view(2)还是regular table(1).
creation_time 字段相同
以下字段 在 TABLE_QUERY() 中不可用，因为它们是 __TABLES__ 但不是 __TABLES_SUMMARY__ 的成员。它们被保留在这里是为了历史利益和部分记录 __TABLES__ metatable:
last_modified_time：时间，自 1970 年 1 月 1 日以来的毫秒数，table 已更新（元数据或 table 内容）。请注意，如果您使用tabledata.insertAll()将记录流式传输到您的 table，这可能会过时几分钟。row_count：table. 中的行数
size_bytes：table. 的总大小（以字节为单位）
您可以使用命令行工具执行此操作
bq show ds_name.table_name
它将显示有关 table 的一些信息，包括 "Total Bytes"。参考这里https://cloud.google.com/bigquery/bq-command-line-tool
或者从 GUI 中，您可以使用内部元数据 table __TABLES__ ，例如，这将为您提供以 GB 为单位的大小：
select
sum(size_bytes)/pow(10,9) as size
from
<your_dataset>.__TABLES__
where
table_id = '<your_table>'
只需使用命令行工具即可。您将看到 table，其中包含 Last-Modified、Schema、Total Rows、Total Bytes 等列。
- 单身Table
bq show dataset_name.table_name
- 所有 Tables
bq show dataset_name.*