执行 sqoop 作业时出现运行时异常

Question

我正在尝试在 biginsights 中执行 sqoop 作业。我正在将数据从 oracle db 导入到 hdfs。下面是开始执行映射器并在一段时间后停止的 sqoop 命令。

sudo sqoop import --connect "jdbc:oracle:thin:@(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)(HOST = ***.**.**.***)(PORT = 1908)) (CONNECT_DATA = (SERVER = DEDICATED) (SERVICE_NAME = GPRSDB1) ) )" --username **** --password ***** --query "select distinct pr_con_id from s_org_ext where $CONDITIONS" --where "PAR_DIVN_ID ='1-1POG9V' AND rownum< 100" --target-dir /user/bigsql/crm/s_contact_mh --fields-terminated-by ',' --m 1

错误如下：

Error: java.lang.RuntimeException: java.lang.RuntimeException: java.sql.SQLException: The Network Adapter could not establish the connection
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:746)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
        at org.apache.hadoop.mapred.YarnChild.run(YarnChild.java:163)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.RuntimeException: java.sql.SQLException: The Network Adapter could not establish the connection
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165)
        ... 9 more
Caused by: java.sql.SQLException: The Network Adapter could not establish the connection
        at oracle.jdbc.driver.SQLStateMapping.newSQLException(SQLStateMapping.java:70)
        at oracle.jdbc.driver.DatabaseError.newSQLException(DatabaseError.java:133)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:199)
        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:480)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:413)
        at oracle.jdbc.driver.PhysicalConnection.<init>(PhysicalConnection.java:508)
        at oracle.jdbc.driver.T4CConnection.<init>(T4CConnection.java:203)
        at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:33)
        at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:510)
        at java.sql.DriverManager.getConnection(DriverManager.java:571)
        at java.sql.DriverManager.getConnection(DriverManager.java:215)
        at org.apache.sqoop.mapreduce.db.DBConfiguration.getConnection(DBConfiguration.java:302)
        at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:213)
        ... 10 more
Caused by: oracle.net.ns.NetException: The Network Adapter could not establish the connection
        at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:328)
        at oracle.net.resolver.AddrResolution.resolveAndExecute(AddrResolution.java:421)
        at oracle.net.ns.NSProtocol.establishConnection(NSProtocol.java:630)
        at oracle.net.ns.NSProtocol.connect(NSProtocol.java:206)
        at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:966)
        at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:292)
        ... 18 more
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at oracle.net.nt.TcpNTAdapter.connect(TcpNTAdapter.java:127)
        at oracle.net.nt.ConnOption.connect(ConnOption.java:126)
        at oracle.net.nt.ConnStrategy.execute(ConnStrategy.java:306)
        ... 23 more

请帮我解决这个问题。提前致谢。

Answer 1

SQOOP Eval 函数只作用于 RDBMS 数据库，returns 你是结果集。 hadoop 在这里没有出现。 SQOOP Import 函数尝试从 RDBMS 导入数据并将其加载到 HDFS。要使 Sqoop Import 正常工作，应该可以从集群中的所有节点访问数据库服务器。

执行 sqoop 作业时出现运行时异常

Runtime exception while executing sqoop job

mapreduce

hadoop-yarn

sqoop

sqoop2

biginsights