无法在 Vertica 中安装 HCatalog 连接器
Not able to install HCatalog Connector in Vertica
我复制了
中的所有 hadoop 文件
/opt/vertica/packages/hcat/lib
现在当我 运行
./hcatUtil --verifyJars --hcatLibPath=/opt/vertica/packages/hcat
我明白了
Summary
-------
Found 27 out of 27 required JARs
Found 5 out of 5 optional Hive SerDe related JARs
Found 2 out of 2 optional native libraries, 5 files in total
Looking at [/opt/vertica/packages/hcat] for configuration files
Found configuration file [cli.xml]
Found configuration file [data-manipulation-statements.xml]
Found configuration file [joins.xml]
Found configuration file [reflect.xml]
Found configuration file [index.xml]
Found configuration file [hdfs-site.xml]
Found configuration file [yarn-site.xml]
Found configuration file [ssl-client.xml]
Found configuration file [core-site.xml]
Found configuration file [project.xml]
Found configuration file [mapred-site.xml]
Found configuration file [var_substitution.xml]
Found configuration file [hive-site.xml]
Found configuration file [working_with_bucketed_tables.xml]
但是当我运行
vsql
\cd /opt/vertica/packages/hcat/ddl
\i install.sql
我收到一个错误
sql:install.sql:13: NOTICE 6564: Found Hadoop configuration files in dependency paths. If any of the configuration files is changed, please re-install HCatalog connector library with modified configuration files
CREATE LIBRARY
vsql:install.sql:16: ROLLBACK 3399: Failure in UDx RPC call InvokeSetExecContext(): Error in User Defined Object [VHCatSource], error code: 0
Couldn't instantiate class com.vertica.hcatalogudl.HCatalogSplitsNoOpSourceFactory
vsql:install.sql:17: ROLLBACK 2059: Source with specified name and parameters does not exist: VHCatSource
vsql:install.sql:18: ROLLBACK 3399: Failure in UDx RPC call InvokeSetExecContext(): Error in User Defined Object [VHCatParser], error code: 0
Couldn't instantiate class com.vertica.hcatalogudl.HCatalogSplitsParserFactory
vsql:install.sql:19: ROLLBACK 2059: Parser with specified name and parameters does not exist: VHCatParser
vsql:install.sql:20: ROLLBACK 3472: Function with same name and number of parameters already exists: get_webhcat_host
vertica=>
我用谷歌搜索并找到了这个线程
https://community.dev.hpe.com/t5/Vertica-Forum/hcatalog-connector/td-p/222733
所以我将 /etc/hadoop/conf
目录的内容从一个 Hadoop 节点复制到我的 vertica 集群的所有节点。
接下来我做了
\i /opt/vertica/packages/hcat/ddl/uninstall.sql
ALTER DATABASE mydb SET JavaClassPathSuffixForUDx='/etc/hadoop-hdfs-conf/';
\i /opt/vertica/packages/hcat/ddl/install.sql
但这并不能解决问题。我仍然收到相同的错误消息。
编辑::
我对这个问题做了更多的研究,我觉得 HP 运送了错误的 vertica-hcatalogudl.jar 产品。这就是为什么这对其他人不起作用。
如果我这样做
create or replace library public.VHCatalogLib as '/opt/vertica/packages/hcat/lib2/vertica-hcatalogudl.jar' DEPENDS '/opt/vertica/packages/hcat/lib/*.*' language 'java';
我可以看到这个库安装了以下对象
SELECT * FROM USER_LIBRARY_MANIFEST WHERE lib_name = 'VHCatalogLib';
schema_name | lib_name | lib_oid | obj_name | obj_type | arg_types | return_type
-------------+--------------+-------------------+--------------------------------------------------+-----------------+-----------+-------------
public | VHCatalogLib | 45035996371219906 | com.vertica.hcatalogudl.WebHCatConfLoaderFactory | Scalar Function | | Varchar
(1 row)
但是当我阅读 hcatalog 连接器附带的 install.sql 的源代码时。它说
CREATE SOURCE public.VHCatSource as LANGUAGE 'java' name 'com.vertica.hcatalogudl.HCatalogSplitsNoOpSourceFactory' library public.VHCatalogLib;
GRANT EXECUTE ON SOURCE public.VHCatSource() TO PUBLIC;
CREATE PARSER public.VHCatParser as LANGUAGE 'java' name 'com.vertica.hcatalogudl.HCatalogSplitsParserFactory' library public.VHCatalogLib;
GRANT EXECUTE ON PARSER public.VHCatParser() TO PUBLIC;
但是库从未安装
- com.vertica.hcatalogudl.HCatalogSplitsNoOpSourceFactory
- com.vertica.hcatalogudl.HCatalogSplitsParserFactory
这就是我们得到错误的原因
无法实例化 class com.vertica.hcatalogudl.HCatalogSplitsParserFactory
因为这是因为他们发送了错误的 vertica-hcatalogudl.jar 文件。
我很确定在 HP vertica 团队的某个地方有一个 vertica-hcatalogudl.jar 文件,其中包含使安装成功所需的所有 3 个对象。
在 HP 支持的帮助下,我能够解决问题。结果在这里
问题的要点是我使用的是 Vertica 7.1.2-2 和 Cloudera 5.4.1,它们彼此不兼容。
将我的 Vertica 实例升级到 7.2 之后......一切都非常顺利。
我复制了
中的所有 hadoop 文件/opt/vertica/packages/hcat/lib
现在当我 运行
./hcatUtil --verifyJars --hcatLibPath=/opt/vertica/packages/hcat
我明白了
Summary
-------
Found 27 out of 27 required JARs
Found 5 out of 5 optional Hive SerDe related JARs
Found 2 out of 2 optional native libraries, 5 files in total
Looking at [/opt/vertica/packages/hcat] for configuration files
Found configuration file [cli.xml]
Found configuration file [data-manipulation-statements.xml]
Found configuration file [joins.xml]
Found configuration file [reflect.xml]
Found configuration file [index.xml]
Found configuration file [hdfs-site.xml]
Found configuration file [yarn-site.xml]
Found configuration file [ssl-client.xml]
Found configuration file [core-site.xml]
Found configuration file [project.xml]
Found configuration file [mapred-site.xml]
Found configuration file [var_substitution.xml]
Found configuration file [hive-site.xml]
Found configuration file [working_with_bucketed_tables.xml]
但是当我运行
vsql
\cd /opt/vertica/packages/hcat/ddl
\i install.sql
我收到一个错误
sql:install.sql:13: NOTICE 6564: Found Hadoop configuration files in dependency paths. If any of the configuration files is changed, please re-install HCatalog connector library with modified configuration files
CREATE LIBRARY
vsql:install.sql:16: ROLLBACK 3399: Failure in UDx RPC call InvokeSetExecContext(): Error in User Defined Object [VHCatSource], error code: 0
Couldn't instantiate class com.vertica.hcatalogudl.HCatalogSplitsNoOpSourceFactory
vsql:install.sql:17: ROLLBACK 2059: Source with specified name and parameters does not exist: VHCatSource
vsql:install.sql:18: ROLLBACK 3399: Failure in UDx RPC call InvokeSetExecContext(): Error in User Defined Object [VHCatParser], error code: 0
Couldn't instantiate class com.vertica.hcatalogudl.HCatalogSplitsParserFactory
vsql:install.sql:19: ROLLBACK 2059: Parser with specified name and parameters does not exist: VHCatParser
vsql:install.sql:20: ROLLBACK 3472: Function with same name and number of parameters already exists: get_webhcat_host
vertica=>
我用谷歌搜索并找到了这个线程 https://community.dev.hpe.com/t5/Vertica-Forum/hcatalog-connector/td-p/222733
所以我将 /etc/hadoop/conf
目录的内容从一个 Hadoop 节点复制到我的 vertica 集群的所有节点。
接下来我做了
\i /opt/vertica/packages/hcat/ddl/uninstall.sql
ALTER DATABASE mydb SET JavaClassPathSuffixForUDx='/etc/hadoop-hdfs-conf/';
\i /opt/vertica/packages/hcat/ddl/install.sql
但这并不能解决问题。我仍然收到相同的错误消息。
编辑::
我对这个问题做了更多的研究,我觉得 HP 运送了错误的 vertica-hcatalogudl.jar 产品。这就是为什么这对其他人不起作用。
如果我这样做
create or replace library public.VHCatalogLib as '/opt/vertica/packages/hcat/lib2/vertica-hcatalogudl.jar' DEPENDS '/opt/vertica/packages/hcat/lib/*.*' language 'java';
我可以看到这个库安装了以下对象
SELECT * FROM USER_LIBRARY_MANIFEST WHERE lib_name = 'VHCatalogLib';
schema_name | lib_name | lib_oid | obj_name | obj_type | arg_types | return_type
-------------+--------------+-------------------+--------------------------------------------------+-----------------+-----------+-------------
public | VHCatalogLib | 45035996371219906 | com.vertica.hcatalogudl.WebHCatConfLoaderFactory | Scalar Function | | Varchar
(1 row)
但是当我阅读 hcatalog 连接器附带的 install.sql 的源代码时。它说
CREATE SOURCE public.VHCatSource as LANGUAGE 'java' name 'com.vertica.hcatalogudl.HCatalogSplitsNoOpSourceFactory' library public.VHCatalogLib;
GRANT EXECUTE ON SOURCE public.VHCatSource() TO PUBLIC;
CREATE PARSER public.VHCatParser as LANGUAGE 'java' name 'com.vertica.hcatalogudl.HCatalogSplitsParserFactory' library public.VHCatalogLib;
GRANT EXECUTE ON PARSER public.VHCatParser() TO PUBLIC;
但是库从未安装
- com.vertica.hcatalogudl.HCatalogSplitsNoOpSourceFactory
- com.vertica.hcatalogudl.HCatalogSplitsParserFactory
这就是我们得到错误的原因
无法实例化 class com.vertica.hcatalogudl.HCatalogSplitsParserFactory
因为这是因为他们发送了错误的 vertica-hcatalogudl.jar 文件。
我很确定在 HP vertica 团队的某个地方有一个 vertica-hcatalogudl.jar 文件,其中包含使安装成功所需的所有 3 个对象。
在 HP 支持的帮助下,我能够解决问题。结果在这里
问题的要点是我使用的是 Vertica 7.1.2-2 和 Cloudera 5.4.1,它们彼此不兼容。
将我的 Vertica 实例升级到 7.2 之后......一切都非常顺利。