如何通过 Eclipse 连接到 EMR 上的 Hive？

Question

我正在尝试通过 Eclipse 连接到 EMR 上的 Hive，但出现错误。

Exception in thread "main" java.sql.SQLException: Could not open client transport with JDBC Uri: jdbc:hive2://localhost:8158: java.net.ConnectException: Connection refused
    at org.apache.hive.jdbc.HiveConnection.openTransport(HiveConnection.java:215)
    at org.apache.hive.jdbc.HiveConnection.<init>(HiveConnection.java:163)
    at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
    at java.sql.DriverManager.getConnection(DriverManager.java:664)
    at java.sql.DriverManager.getConnection(DriverManager.java:247)
    at com.readypulse.sparkanalytics.HiveQLConnector.<init>(HiveQLConnector.java:31)
    at com.readypulse.sparkanalytics.HiveQLConnector.main(HiveQLConnector.java:83)
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused

Answer 1

您需要一个 SSH 隧道来为 EMR 主机打个洞。然后通过 JDBC.

连接

private static String driverName = "org.apache.hive.jdbc.HiveDriver";
private static String hiveConnectionString = "jdbc:hive2://localhost:8158";

//Need port forwarding
SparkPortForwarding.portForwardForSpark();

Class c = Class.forName(driverName);
Connection connection = DriverManager.getConnection(hiveConnectionString,
                "user", "pwd");
Statement stmt = connection.createStatement();
ResultSet rs = stmt.executeQuery(sql);

您可以通过 shell 进行端口转发：

ssh -i ~/mykey.pem -N -L 8158:<maserhost>:10000 hadoop@<masterhost>

或者您可以使用库 JSch 中的代码

public static void portForwardForSpark() {
    try {
        if(session != null && session.isConnected()) {
            return;
        }

        JSch jsch = new JSch();
        jsch.addIdentity(PATH_TO_SSH_KEY_PEM);
        String host = REMOTE_HOST;
        session = jsch.getSession(USER, host, 22);

        // username and password will be given via UserInfo interface.
        UserInfo ui = new MyUserInfo();
        session.setUserInfo(ui);

        session.connect();
        int assingedPort = session.setPortForwardingL(LPORT, RHOST, RPORT);
        System.out.println("Port forwarding done for the post : " + assingedPort);
    } catch (Exception e) {
        System.out.println(e);
    }
}

如何通过 Eclipse 连接到 EMR 上的 Hive？

How do I connect to Hive on EMR through Eclipse?

eclipse

hive

amazon-web-services

amazon-emr

emr