Apache Drill S3 存储 API

Question

我们正在尝试使用 Apache Drill Rest 创建一个存储插件 API。我有一个列出我的配置的文件：

**s3.config**                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                     
{
  "type": "file",
  "enabled": true,
  "connection": "s3a://AWS_ACCESS_KEY_ID:AWS_SECRET_KEY@BUCKET_NAME",
  "config": {
    "fs.s3a.access.key": "AWS_ACCESS_KEY_ID",
    "fs.s3a.secret.key": "AWS_SECRET_KEY"
  },
  "workspaces": {
    "root": {
      "location": "/data",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    },
    "tmp": {
      "location": "/tmp",
      "writable": true,
      "defaultInputFormat": null,
      "allowAccessOutsideWorkspace": false
    }
  },
  "formats": {
    "parquet": {
      "type": "parquet"
    }
  }
}

当我 post 此数据使用 curl against drill API 时，以下是我收到的错误：

Unexpected token (END_OBJECT), expected FIELD_NAME: missing property 'type' that is to contain type id  (for class org.apache.drill.common.logical.StoragePluginConfig)

我的 curl 请求如下所示：

curl -v --write-out %{http_code} -H "Content-Type: application/json" -X POST  --data "@s3.config" http://localhost:8047/storage/s3.json

Answer 1

你有错误的请求正文，你错过了 "name" 插件名称属性和 "config" 属性本身。试试这个配置：

{
  "name":"s3",
  "config": {
    "type": "file",
    "enabled": true,
    "connection": "s3a://AWS_ACCESS_KEY_ID:AWS_SECRET_KEY@BUCKET_NAME",
    "config": {
      "fs.s3a.access.key": "AWS_ACCESS_KEY_ID",
      "fs.s3a.secret.key": "AWS_SECRET_KEY"
    },
    "workspaces": {
      "root": {
        "location": "/data",
        "writable": true,
        "defaultInputFormat": null,
        "allowAccessOutsideWorkspace": false
      },
      "tmp": {
        "location": "/tmp",
        "writable": true,
        "defaultInputFormat": null,
        "allowAccessOutsideWorkspace": false
      }
    },
    "formats": {
      "parquet": {
        "type": "parquet"
      }
    }
  }
}

查看更多： https://drill.apache.org/docs/rest-api-introduction/#post-/storage/{name}.json

Apache Drill S3 存储 API

Apache Drill S3 storage with API

json

curl

amazon-s3

apache-drill