java.sql.SQLException：对 SQOOP 导入抛出期望

Question

当我尝试使用

将存储在 mysql 数据库中的 table 存储到我的 HDFS 中时

sqoop import --connect jdbc:mysql://hostname1.com/mydb --username user1 --password pwd1 --table emp1;

我遇到以下异常：

Warning: /opt/cloudera/parcels/CDH-5.4.3-1.cdh5.4.3.p0.6/bin/../lib/sqoop/../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
15/07/24 22:48:54 INFO sqoop.Sqoop: Running Sqoop version: 1.4.5-cdh5.4.3
15/07/24 22:48:54 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
15/07/24 22:48:55 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
15/07/24 22:48:55 INFO tool.CodeGenTool: Beginning code generation
15/07/24 22:48:55 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `emp1` AS t LIMIT 1
15/07/24 22:48:55 ERROR manager.SqlManager: Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@2d749418 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@2d749418 is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:934)
        at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:931)
        at com.mysql.jdbc.MysqlIO.checkForOutstandingStreamingData(MysqlIO.java:2735)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1899)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2619)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2569)
        at com.mysql.jdbc.StatementImpl.executeQuery(StatementImpl.java:1524)
        at com.mysql.jdbc.ConnectionImpl.getMaxBytesPerChar(ConnectionImpl.java:3003)
        at com.mysql.jdbc.Field.getMaxBytesPerCharacter(Field.java:602)
        at com.mysql.jdbc.ResultSetMetaData.getPrecision(ResultSetMetaData.java:445)
        at org.apache.sqoop.manager.SqlManager.getColumnInfoForRawQuery(SqlManager.java:286)
        at org.apache.sqoop.manager.SqlManager.getColumnTypesForRawQuery(SqlManager.java:241)
        at org.apache.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:227)
        at org.apache.sqoop.manager.ConnManager.getColumnTypes(ConnManager.java:295)
        at org.apache.sqoop.orm.ClassWriter.getColumnTypes(ClassWriter.java:1833)
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1645)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
15/07/24 22:48:55 ERROR tool.ImportTool: Encountered IOException running import job: java.io.IOException: No columns to generate for ClassWriter
        at org.apache.sqoop.orm.ClassWriter.generate(ClassWriter.java:1651)
        at org.apache.sqoop.tool.CodeGenTool.generateORM(CodeGenTool.java:96)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:478)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)

我的sqoop版本是运行 Sqoop版本：1.4.5-cdh5.4.3和mysql-connector版本是5.1.31。

对修复我的异常有帮助吗？提前致谢。

Answer 1

在这个版本的 Sqoop 中，获取大小存在一些问题，在 MySQLManager class 中被设置为 Integer.MIN_VALUE (-2147483648) /src/java/org/apache/sqoop/manager/MySQLManager.java。默认值集是 0，在此 class 中被覆盖。因此，与该行关联的结果集不会释放锁，因此不会释放错误消息。请参阅 Sqoop 源代码中的以下代码：

@Override
  protected void initOptionDefaults() {
    if (options.getFetchSize() == null) {
      LOG.info("Preparing to use a MySQL streaming resultset.");
      options.setFetchSize(Integer.MIN_VALUE);      //this line is causing the error
    } else if (
        !options.getFetchSize().equals(Integer.MIN_VALUE)
        && !options.getFetchSize().equals(0)) {
      LOG.info("Argument '--fetch-size " + options.getFetchSize()
          + "' will probably get ignored by MySQL JDBC driver.");
      // see also
      // http://dev.mysql.com/doc/refman/5.5/en
      //                       /connector-j-reference-implementation-notes.html
    }
  }

要解决此错误，请从您的 Sqoop 版本的源代码的 MySQLManager class 中删除上述 initOptionDefaults 方法。

REF:https://issues.apache.org/jira/browse/SQOOP-1400

Answer 2

如果您使用的是 Sqoop 客户端 Java API，您可以在 SqoopOptions 中明确指定提取大小：

private static SqoopOptions SqoopOptions = new SqoopOptions();
 SqoopOptions.setFetchSize(1000);

java.sql.SQLException：对 SQOOP 导入抛出期望

java.sql.SQLException: expection being thrown for SQOOP import

mysql

hadoop

mapreduce

jdbc

sqoop