改变镶木地板日志级别

Question

我在 java 程序中使用 org.apache.parquet 将 json 文件转换为 parquet 格式。但是，无论我尝试什么，我都无法禁用镶木地板自己的日志记录到标准输出。有什么方法可以改变 parquet 日志记录级别，或者完全关闭它吗？

stdout 上的日志消息示例...

12-Feb-2017 18:12:21 INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 427B for [verb] BINARY: 2,890 values, 385B raw, 390B comp, 1 pages, encodings: [BIT_PACKED, PLAIN_DICTIONARY], dic { 2 entries, 17B raw, 2B comp}
12-Feb-2017 18:12:21 INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 3,256B for [postedTime] BINARY: 2,890 values, 3,585B raw, 3,180B comp, 1 pages, encodings: [BIT_PACKED, PLAIN_DICTIONARY], dic { 593 entries, 16,604B raw, 593B comp}
12-Feb-2017 18:12:21 INFO: org.apache.parquet.hadoop.ColumnChunkPageWriteStore: written 4,611B for [message] BINARY: 2,890 values, 4,351B raw, 4,356B comp, 1 pages, encodings: [BIT_PACKED, PLAIN_DICTIONARY], dic { 2,088 entries, 263,329B raw, 2,088B comp}

我如何称呼 parquet 的示例...

public void writeToParquet(List<GenericData.Record> recordsToWrite, Path fileToWrite) throws IOException {
    try (ParquetWriter<GenericData.Record> writer = AvroParquetWriter
            .<GenericData.Record>builder(fileToWrite)
            .withSchema(SCHEMA)
            .withConf(new Configuration())
            .withCompressionCodec(CompressionCodecName.SNAPPY)
            .build()) {

        for (GenericData.Record record : recordsToWrite) {
            writer.write(record);
        }
    }
}

Answer 1

我知道这是一个老问题，但我刚刚在 CDH 5.x 中将 Parquet 与 Hive 结合使用时遇到了这个问题并找到了解决方法。看这里：

也许其他人会觉得它有用。

改变镶木地板日志级别

Alter parquet log level

java

parquet