如何修复 AWS EMR 中的 not 运行 presto 插件
How to fix not running presto plugin in AWS EMR
关于
我正在尝试使用像 wyukawa/presto-fluentd 这样的 presto 插件，它适用于 localhost(mac os x)，但不适用于亚马逊电子病历
详情
本地ost
起初，我尝试在 localhost(mac os x) 上工作，但它确实有效。
插件目录
reizist ...plugin/presto-fluentd $ pwd
/usr/local/Cellar/presto/0.185/libexec/plugin/presto-fluentd
reizist ...plugin/presto-fluentd $ ls -1
fluency-1.3.0.jar
guava-21.0.jar
jackson-annotations-2.8.1.jar
jackson-core-2.7.1.jar
jackson-databind-2.7.1.jar
jackson-dataformat-msgpack-0.8.12.jar
jolokia-jvm-1.3.7-agent.jar
log-0.148.jar
msgpack-core-0.8.12.jar
phi-accural-failure-detector-0.0.4.jar
presto-fluentd-0.0.1.jar
slf4j-api-1.7.22.jar
属性
reizist ...libexec/etc $ pwd
/usr/local/Cellar/presto/0.185/libexec/etc
reizist ...libexec/etc $ ls -1
catalog
config.properties
event-listener.properties
jvm.config
log.properties
node.properties
reizist ...libexec/etc $ cat event-listener.properties
event-listener.name=presto-fluentd
event-listener.fluentd-host=localhost
event-listener.fluentd-port=24224
event-listener.fluentd-tag=presto.query
left: presto log, center: fluentd log, right: presto-cli
电子病历
我也在 EMR 上的 EC2 上尝试了同样的方法，但它没有用。
That plugin is correctly loaded, event listener registered，所以觉得奇怪。
插件目录
[hadoop@ip-172-31-29-54 plugin]$ pwd
/usr/lib/presto/plugin
[hadoop@ip-172-31-29-54 plugin]$ ls
accumulo cassandra jmx memory mysql redis tpch
atop example-http kafka ml postgresql resource-group-managers
blackhole hive-hadoop2 localfile mongodb presto-fluentd teradata-functions
[hadoop@ip-172-31-29-54 plugin]$ ls -1 presto-fluentd/
fluency-1.3.0.jar
guava-21.0.jar
jackson-annotations-2.8.1.jar
jackson-core-2.7.1.jar
jackson-databind-2.7.1.jar
jackson-dataformat-msgpack-0.8.12.jar
log-0.148.jar
msgpack-core-0.8.12.jar
phi-accural-failure-detector-0.0.4.jar
presto-fluentd-0.0.1.jar
slf4j-api-1.7.22.jar
属性
[hadoop@ip-172-31-29-54 presto]$ pwd
/etc/presto
[hadoop@ip-172-31-29-54 presto]$ tree .
.
├── conf -> /etc/alternatives/presto-conf
├── conf.dist
│ ├── catalog
│ │ ├── hive.properties
│ │ └── mysql.properties
│ ├── config.properties
│ ├── jvm.config
│ ├── log.properties
│ ├── node.properties
│ └── presto-env.sh
└── event-listener.properties
3 directories, 8 files
[hadoop@ip-172-31-29-54 presto]$ cat event-listener.properties
event-listener.name=presto-fluentd
event-listener.fluentd-host=localhost
event-listener.fluentd-port=24224
event-listener.fluentd-tag=presto.query
我也通过插入打印调试代码进行了测试，但好像没有加载。
我应该如何在 EMR 上使用这个插件？
谢谢。
补充
这里是 fluentd 配置。
<source>
@type forward
</source>
<match *.**>
@type stdout
</match>
我自己解决了
实际上我必须在 /mnt/var/lib/presto/data/etc 上找到 event-listener.properties，所以我这样做了：
$s3uri="s3://my-s3-bucket"
# make symbolic link
sudo mkdir /usr/lib/presto/etc
sudo ln -s /usr/lib/presto/etc /mnt/var/lib/presto/data
# download presto plugins
aws s3 sync $s3uri/jar/ /usr/lib/presto/plugin/
aws s3 sync $s3uri/properties/ /usr/lib/presto/etc/
# make sure all plugins are owned by presto user
chown -R presto:presto /usr/lib/presto/plugin
chown -R presto:presto /usr/lib/presto/etc
# restart presto
stop presto-server
start presto-server
最后我的目录如下：
[hadoop@ip-172-31-21-25 presto]$ pwd
/usr/lib/presto
[hadoop@ip-172-31-21-25 presto]$ ls -alh
total 228K
drwxr-xr-x 7 root root 4.0K Oct 19 06:52 .
dr-xr-xr-x 47 root root 4.0K Oct 19 06:30 ..
drwxr-xr-x 3 presto presto 4.0K Oct 19 06:30 bin
drwxr-xr-x 3 presto presto 4.0K Oct 19 06:52 etc
drwxr-xr-x 2 presto presto 12K Oct 19 06:30 lib
-rw-r--r-- 1 presto presto 188K Sep 22 22:54 NOTICE
drwxr-xr-x 24 presto presto 4.0K Oct 19 06:45 plugin
drwxr-xr-x 2 presto presto 4.0K Oct 19 06:30 presto-jdbc
-rw-r--r-- 1 presto presto 119 Sep 22 22:54 README.txt
[hadoop@ip-172-31-21-25 etc]$ pwd
/usr/lib/presto/etc
[hadoop@ip-172-31-21-25 etc]$ ls
event-listener.properties
[hadoop@ip-172-31-21-25 plugin]$ pwd
/usr/lib/presto/plugin
[hadoop@ip-172-31-21-25 plugin]$ ls
accumulo example-http localfile mysql redis tpcds
atop hive-hadoop2 memory postgresql resource-group-managers tpch
blackhole jmx ml presto-fluentd sqlserver
cassandra kafka mongodb presto-thrift teradata-functions
并且它工作正常。
关于
我正在尝试使用像 wyukawa/presto-fluentd 这样的 presto 插件，它适用于 localhost(mac os x)，但不适用于亚马逊电子病历
详情
本地ost
起初，我尝试在 localhost(mac os x) 上工作，但它确实有效。
插件目录
reizist ...plugin/presto-fluentd $ pwd /usr/local/Cellar/presto/0.185/libexec/plugin/presto-fluentd reizist ...plugin/presto-fluentd $ ls -1 fluency-1.3.0.jar guava-21.0.jar jackson-annotations-2.8.1.jar jackson-core-2.7.1.jar jackson-databind-2.7.1.jar jackson-dataformat-msgpack-0.8.12.jar jolokia-jvm-1.3.7-agent.jar log-0.148.jar msgpack-core-0.8.12.jar phi-accural-failure-detector-0.0.4.jar presto-fluentd-0.0.1.jar slf4j-api-1.7.22.jar属性
reizist ...libexec/etc $ pwd /usr/local/Cellar/presto/0.185/libexec/etc reizist ...libexec/etc $ ls -1 catalog config.properties event-listener.properties jvm.config log.properties node.properties reizist ...libexec/etc $ cat event-listener.properties event-listener.name=presto-fluentd event-listener.fluentd-host=localhost event-listener.fluentd-port=24224 event-listener.fluentd-tag=presto.query
left: presto log, center: fluentd log, right: presto-cli
电子病历
我也在 EMR 上的 EC2 上尝试了同样的方法，但它没有用。 That plugin is correctly loaded, event listener registered，所以觉得奇怪。
插件目录
[hadoop@ip-172-31-29-54 plugin]$ pwd /usr/lib/presto/plugin [hadoop@ip-172-31-29-54 plugin]$ ls accumulo cassandra jmx memory mysql redis tpch atop example-http kafka ml postgresql resource-group-managers blackhole hive-hadoop2 localfile mongodb presto-fluentd teradata-functions [hadoop@ip-172-31-29-54 plugin]$ ls -1 presto-fluentd/ fluency-1.3.0.jar guava-21.0.jar jackson-annotations-2.8.1.jar jackson-core-2.7.1.jar jackson-databind-2.7.1.jar jackson-dataformat-msgpack-0.8.12.jar log-0.148.jar msgpack-core-0.8.12.jar phi-accural-failure-detector-0.0.4.jar presto-fluentd-0.0.1.jar slf4j-api-1.7.22.jar属性
[hadoop@ip-172-31-29-54 presto]$ pwd /etc/presto [hadoop@ip-172-31-29-54 presto]$ tree . . ├── conf -> /etc/alternatives/presto-conf ├── conf.dist │ ├── catalog │ │ ├── hive.properties │ │ └── mysql.properties │ ├── config.properties │ ├── jvm.config │ ├── log.properties │ ├── node.properties │ └── presto-env.sh └── event-listener.properties 3 directories, 8 files [hadoop@ip-172-31-29-54 presto]$ cat event-listener.properties event-listener.name=presto-fluentd event-listener.fluentd-host=localhost event-listener.fluentd-port=24224 event-listener.fluentd-tag=presto.query
我也通过插入打印调试代码进行了测试，但好像没有加载。 我应该如何在 EMR 上使用这个插件？
谢谢。
补充
这里是 fluentd 配置。
<source>
@type forward
</source>
<match *.**>
@type stdout
</match>
我自己解决了
实际上我必须在 /mnt/var/lib/presto/data/etc 上找到 event-listener.properties，所以我这样做了：
$s3uri="s3://my-s3-bucket"
# make symbolic link
sudo mkdir /usr/lib/presto/etc
sudo ln -s /usr/lib/presto/etc /mnt/var/lib/presto/data
# download presto plugins
aws s3 sync $s3uri/jar/ /usr/lib/presto/plugin/
aws s3 sync $s3uri/properties/ /usr/lib/presto/etc/
# make sure all plugins are owned by presto user
chown -R presto:presto /usr/lib/presto/plugin
chown -R presto:presto /usr/lib/presto/etc
# restart presto
stop presto-server
start presto-server
最后我的目录如下：
[hadoop@ip-172-31-21-25 presto]$ pwd
/usr/lib/presto
[hadoop@ip-172-31-21-25 presto]$ ls -alh
total 228K
drwxr-xr-x 7 root root 4.0K Oct 19 06:52 .
dr-xr-xr-x 47 root root 4.0K Oct 19 06:30 ..
drwxr-xr-x 3 presto presto 4.0K Oct 19 06:30 bin
drwxr-xr-x 3 presto presto 4.0K Oct 19 06:52 etc
drwxr-xr-x 2 presto presto 12K Oct 19 06:30 lib
-rw-r--r-- 1 presto presto 188K Sep 22 22:54 NOTICE
drwxr-xr-x 24 presto presto 4.0K Oct 19 06:45 plugin
drwxr-xr-x 2 presto presto 4.0K Oct 19 06:30 presto-jdbc
-rw-r--r-- 1 presto presto 119 Sep 22 22:54 README.txt
[hadoop@ip-172-31-21-25 etc]$ pwd
/usr/lib/presto/etc
[hadoop@ip-172-31-21-25 etc]$ ls
event-listener.properties
[hadoop@ip-172-31-21-25 plugin]$ pwd
/usr/lib/presto/plugin
[hadoop@ip-172-31-21-25 plugin]$ ls
accumulo example-http localfile mysql redis tpcds
atop hive-hadoop2 memory postgresql resource-group-managers tpch
blackhole jmx ml presto-fluentd sqlserver
cassandra kafka mongodb presto-thrift teradata-functions
并且它工作正常。