如何在配置单元中一次删除所有分区?
How do I drop all partitions at once in hive?
Hive 版本 1.1
我有一个外部配置单元 table,如下所示:
CREATE EXTERNAL TABLE `schedule_events`(
`schedule_id` string COMMENT 'from deserializer',
`service_key` string COMMENT 'from deserializer',
`event_start_date_time` string COMMENT 'from deserializer',
`event_id` string COMMENT 'from deserializer',
`event_type` string COMMENT 'from deserializer',
`transitional_key` string COMMENT 'from deserializer',
`created_date_time` string COMMENT 'from deserializer',
`bus_date` string COMMENT 'from deserializer')
PARTITIONED BY (
`year` string,
`month` string,
`day` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION
'hdfs://nameservice1/hadoop/raw/omega/scheduled_events'
TBLPROPERTIES (
'avro.schema.url'='hdfs:////hadoop/raw/omega/schema/schedule_events.avsc',
'transient_lastDdlTime'='1505742141')
现在要删除特定分区,我可以 运行 如下所示的 ALTER 命令
ALTER TABLE schedule_events DROP IF EXISTS PARTITION (year='2016',month='06',day='01')
Dropped the partition year=2016/month=06/day=01
hive> show partitions schedule_events;
OK
year=2017/month=09/day=01
year=2017/month=09/day=02
year=2017/month=09/day=03
year=2017/month=09/day=04
year=2017/month=09/day=05
但是这个 table 有很多分区。
如何一次删除所有现有分区?我想一次删除所有现有分区?这可能吗?
有多个选项,这里是一个:
alter table schedule_events drop if exists partition (year<>'');
Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators
"... To drop a partition from a Hive table, this works:
ALTER TABLE foo DROP PARTITION(ds = 'date')
...but it should also work to drop all partitions prior to date.
ALTER TABLE foo DROP PARTITION(ds < 'date')
This task is to implement ALTER TABLE DROP PARTITION for all of the
comparators, < > <= >= <> = != instead of just for ="
您可以使用类似这样的东西:
ALTER TABLE schedule_events drop if exists partition (year>'0');
改变table schema_name.table_name删除分区(partition_column != '');
使用火花 sql:
val paritions_values = spark.sql("show partitions "+databasename+'.'+tablename)
.collect().map(f=>f(0).toString)
.toArray.mkString("partition(", "," , "\")")
.replace("," , "\") ,partition(")
.replace("=", "=\"")
spark.sql("alter table "+databasename+'.'+tablename+" drop "+paritions_values)
Hive 版本 1.1
我有一个外部配置单元 table,如下所示:
CREATE EXTERNAL TABLE `schedule_events`(
`schedule_id` string COMMENT 'from deserializer',
`service_key` string COMMENT 'from deserializer',
`event_start_date_time` string COMMENT 'from deserializer',
`event_id` string COMMENT 'from deserializer',
`event_type` string COMMENT 'from deserializer',
`transitional_key` string COMMENT 'from deserializer',
`created_date_time` string COMMENT 'from deserializer',
`bus_date` string COMMENT 'from deserializer')
PARTITIONED BY (
`year` string,
`month` string,
`day` string)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
LOCATION
'hdfs://nameservice1/hadoop/raw/omega/scheduled_events'
TBLPROPERTIES (
'avro.schema.url'='hdfs:////hadoop/raw/omega/schema/schedule_events.avsc',
'transient_lastDdlTime'='1505742141')
现在要删除特定分区,我可以 运行 如下所示的 ALTER 命令
ALTER TABLE schedule_events DROP IF EXISTS PARTITION (year='2016',month='06',day='01')
Dropped the partition year=2016/month=06/day=01
hive> show partitions schedule_events;
OK
year=2017/month=09/day=01
year=2017/month=09/day=02
year=2017/month=09/day=03
year=2017/month=09/day=04
year=2017/month=09/day=05
但是这个 table 有很多分区。
如何一次删除所有现有分区?我想一次删除所有现有分区?这可能吗?
有多个选项,这里是一个:
alter table schedule_events drop if exists partition (year<>'');
Hive: Extend ALTER TABLE DROP PARTITION syntax to use all comparators
"... To drop a partition from a Hive table, this works:
ALTER TABLE foo DROP PARTITION(ds = 'date')
...but it should also work to drop all partitions prior to date.
ALTER TABLE foo DROP PARTITION(ds < 'date') This task is to implement ALTER TABLE DROP PARTITION for all of the comparators, < > <= >= <> = != instead of just for ="
您可以使用类似这样的东西:
ALTER TABLE schedule_events drop if exists partition (year>'0');
改变table schema_name.table_name删除分区(partition_column != '');
使用火花 sql:
val paritions_values = spark.sql("show partitions "+databasename+'.'+tablename)
.collect().map(f=>f(0).toString)
.toArray.mkString("partition(", "," , "\")")
.replace("," , "\") ,partition(")
.replace("=", "=\"")
spark.sql("alter table "+databasename+'.'+tablename+" drop "+paritions_values)