使用加密密码文件执行 Sqoop 导入时获取 BadPaddingException
Getting BadPaddingException while doing a Sqoop import with encrypted password file
我用 openssl 加密文件然后把它放在 HDFS 上，我使用 AES/ECB，128 位和 salt 选项，通过一些研究我发现 openssl 使用 PKCS5 填充作为默认值，这些都是默认为 CryptoFileLoader class。这是我的加密过程：
# echo -n "password" > .pw
# openssl enc -aes-128-ecb -salt -in .pw -out .pw.enc
# hdfs dfs -put .pw.enc /user/user1/
Sqoop 版本为 1.4.6
命令：
sqoop import \
-Dorg.apache.sqoop.credentials.loader.class=org.apache.sqoop.util.password.CryptoFileLoader \
-Dorg.apache.sqoop.credentials.loader.crypto.passphrase=sqoop \
--connect jdbc:oracle:thin:@host/database \
--username user1 \
--password-file /user/user1/.pw.enc \
--table db.table1 \
--hive-import \
--hive-overwrite \
--hive-table hivedb.table1 \
--hive-drop-import-delims
给出：
17/03/08 15:10:37 WARN tool.BaseSqoopTool: Failed to load password file
java.io.IOException: Can't decrypt the password
at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
at org.apache.sqoop.util.CredentialsUtil.fetchPasswordFromLoader(CredentialsUtil.java:81)
at org.apache.sqoop.util.CredentialsUtil.fetchPassword(CredentialsUtil.java:66)
at org.apache.sqoop.tool.BaseSqoopTool.applyCredentialsOptions(BaseSqoopTool.java:1042)
at org.apache.sqoop.tool.BaseSqoopTool.applyCommonOptions(BaseSqoopTool.java:997)
at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:875)
at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
at org.apache.sqoop.Sqoop.run(Sqoop.java:131)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:966)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:436)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:149)
... 12 more
Error while loading password file: Can't decrypt the password
我也尝试手动提供其他 CryptoFileLoader 参数，并将本地文件传递给 --password-file。
我可以用 openssl 成功解密文件。我无法用 Java 程序解密(?)
我看到填充有问题，但我不知道它是什么以及如何使用某种填充方法或其他任何方法加密文件，我没有加密经验。
class 中还有一个 org.apache.sqoop.credentials.loader.crypto.iterations 参数，它表示 PBKDF2 迭代次数，但我不知道它是否改变了什么。
感谢您的帮助。
我不是 Sqoop 和 Hadoop 的专家，但从你的例外开始
CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
的源码
在我看来，事情与你做的有点不同：密码存储在使用PBKDF2算法的加密文件中，这不等同于应用AES -128-欧洲央行。来自 wikipedia:
PBKDF2 applies a pseudorandom function, such as hash-based message authentication code (HMAC), to the input password or passphrase along with a salt value and repeats the process many times to produce a derived key, which can then be used as a cryptographic key in subsequent operations. The added computational work makes password cracking much more difficult, and is known as key stretching.
无法从 Openssl 命令行执行 PBKDF2。我用Java做了一个小测试，它可能是一个替代
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.NoSuchAlgorithmException;
import javax.crypto.Cipher;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;
public class Test {
/*Default is AES in electronic code book with padding.*/
private static String DEFAULT_ALG = "AES/ECB/PKCS5Padding";
/*Default salt is not much secure, use your own!*/
private static String DEFAULT_SALT = "SALT";
/*Iterate 10000 times by default.*/
private static int DEFAULT_ITERATIONS = 10000;
/*One of valid key sizes for default algorithm (AES).*/
private static int DEFAULT_KEY_LEN = 128;
public static void main(String[] args) throws IOException {
String inputFileName = "C:\temp\in.txt"; /*Enter your input (plain) file path */
String outputFileName = "C:\temp\out.bin"; /*Enter your output (encrypted) file path */
String passPhrase = "mypassphrase"; /*Enter your passphrase */
String salt = DEFAULT_SALT;
String alg = DEFAULT_ALG;
int iterations = DEFAULT_ITERATIONS;
int keyLen = DEFAULT_KEY_LEN;
SecretKeyFactory factory = null;
try {
factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
} catch (NoSuchAlgorithmException e) {
throw new IOException("Can't load SecretKeyFactory", e);
}
SecretKeySpec key = null;
try {
String algOnly = alg.split("/")[0];
key = new SecretKeySpec(
factory.generateSecret(
new PBEKeySpec(passPhrase.toCharArray(), salt.getBytes(), iterations, keyLen)).getEncoded(),
algOnly);
} catch (Exception e) {
throw new IOException("Can't generate secret key", e);
}
Cipher crypto = null;
try {
crypto = Cipher.getInstance(alg);
} catch (Exception e) {
throw new IOException("Can't initialize the decryptor", e);
}
Path inputFileLocation = Paths.get(inputFileName);
byte[] decrypted = Files.readAllBytes(inputFileLocation);
byte[] encrypted;
try {
crypto.init(Cipher.ENCRYPT_MODE, key);
encrypted = crypto.doFinal(decrypted);
} catch (Exception e) {
throw new IOException("Can't decrypt the password", e);
}
Path outputFileLocation = Paths.get(outputFileName);
Files.write(outputFileLocation, encrypted);
}
}
正如 Simone 的回答 - openssl 和 java 实现之间的加密算法有所不同。这就是为什么您可以毫无问题地使用 openssl 进行解密（因为它再次调用自己的（不同的）算法）。
经过大量挖掘，我从 (Dave Thompson) 中找到了这个答案，其中指出：
Short answer: what openssl enc (without -K for raw) uses is not PBKDF2; it is almost PBKDF1, with iteration count 1.
似乎有两种方法可以解决这个问题，要么：
a) 在 java 中找到可以解密 openssl 正在做什么的东西 - 在这个 [=12] 中的 post 中引用了一个 java 库 'BouncyCastle' =]（如果您喜欢使用它而不是标准的 CryptoFile），他们实现了与 openssl 使用的算法完全相同的算法。
或
b) 找到一些其他的命令行实用程序来代替实现 PBKDF2 的 openssl。 nabble.com posting 中还提到了一些不同语言的实现。
（由于引用了 Dave 的关键观察结果）
我用 openssl 加密文件然后把它放在 HDFS 上，我使用 AES/ECB，128 位和 salt 选项，通过一些研究我发现 openssl 使用 PKCS5 填充作为默认值，这些都是默认为 CryptoFileLoader class。这是我的加密过程：
# echo -n "password" > .pw
# openssl enc -aes-128-ecb -salt -in .pw -out .pw.enc
# hdfs dfs -put .pw.enc /user/user1/
Sqoop 版本为 1.4.6
命令：
sqoop import \
-Dorg.apache.sqoop.credentials.loader.class=org.apache.sqoop.util.password.CryptoFileLoader \
-Dorg.apache.sqoop.credentials.loader.crypto.passphrase=sqoop \
--connect jdbc:oracle:thin:@host/database \
--username user1 \
--password-file /user/user1/.pw.enc \
--table db.table1 \
--hive-import \
--hive-overwrite \
--hive-table hivedb.table1 \
--hive-drop-import-delims
给出：
17/03/08 15:10:37 WARN tool.BaseSqoopTool: Failed to load password file
java.io.IOException: Can't decrypt the password
at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
at org.apache.sqoop.util.CredentialsUtil.fetchPasswordFromLoader(CredentialsUtil.java:81)
at org.apache.sqoop.util.CredentialsUtil.fetchPassword(CredentialsUtil.java:66)
at org.apache.sqoop.tool.BaseSqoopTool.applyCredentialsOptions(BaseSqoopTool.java:1042)
at org.apache.sqoop.tool.BaseSqoopTool.applyCommonOptions(BaseSqoopTool.java:997)
at org.apache.sqoop.tool.ImportTool.applyOptions(ImportTool.java:875)
at org.apache.sqoop.tool.SqoopTool.parseArguments(SqoopTool.java:435)
at org.apache.sqoop.Sqoop.run(Sqoop.java:131)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
Caused by: javax.crypto.BadPaddingException: Given final block not properly padded
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:966)
at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:824)
at com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:436)
at javax.crypto.Cipher.doFinal(Cipher.java:2165)
at org.apache.sqoop.util.password.CryptoFileLoader.loadPassword(CryptoFileLoader.java:149)
... 12 more
Error while loading password file: Can't decrypt the password
我也尝试手动提供其他 CryptoFileLoader 参数，并将本地文件传递给 --password-file。
我可以用 openssl 成功解密文件。我无法用 Java 程序解密(?)
我看到填充有问题，但我不知道它是什么以及如何使用某种填充方法或其他任何方法加密文件，我没有加密经验。
class 中还有一个 org.apache.sqoop.credentials.loader.crypto.iterations 参数，它表示 PBKDF2 迭代次数，但我不知道它是否改变了什么。
感谢您的帮助。
我不是 Sqoop 和 Hadoop 的专家，但从你的例外开始
CryptoFileLoader.loadPassword(CryptoFileLoader.java:151)
在我看来，事情与你做的有点不同：密码存储在使用PBKDF2算法的加密文件中，这不等同于应用AES -128-欧洲央行。来自 wikipedia:
PBKDF2 applies a pseudorandom function, such as hash-based message authentication code (HMAC), to the input password or passphrase along with a salt value and repeats the process many times to produce a derived key, which can then be used as a cryptographic key in subsequent operations. The added computational work makes password cracking much more difficult, and is known as key stretching.
无法从 Openssl 命令行执行 PBKDF2。我用Java做了一个小测试，它可能是一个替代
import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.security.NoSuchAlgorithmException;
import javax.crypto.Cipher;
import javax.crypto.SecretKeyFactory;
import javax.crypto.spec.PBEKeySpec;
import javax.crypto.spec.SecretKeySpec;
public class Test {
/*Default is AES in electronic code book with padding.*/
private static String DEFAULT_ALG = "AES/ECB/PKCS5Padding";
/*Default salt is not much secure, use your own!*/
private static String DEFAULT_SALT = "SALT";
/*Iterate 10000 times by default.*/
private static int DEFAULT_ITERATIONS = 10000;
/*One of valid key sizes for default algorithm (AES).*/
private static int DEFAULT_KEY_LEN = 128;
public static void main(String[] args) throws IOException {
String inputFileName = "C:\temp\in.txt"; /*Enter your input (plain) file path */
String outputFileName = "C:\temp\out.bin"; /*Enter your output (encrypted) file path */
String passPhrase = "mypassphrase"; /*Enter your passphrase */
String salt = DEFAULT_SALT;
String alg = DEFAULT_ALG;
int iterations = DEFAULT_ITERATIONS;
int keyLen = DEFAULT_KEY_LEN;
SecretKeyFactory factory = null;
try {
factory = SecretKeyFactory.getInstance("PBKDF2WithHmacSHA1");
} catch (NoSuchAlgorithmException e) {
throw new IOException("Can't load SecretKeyFactory", e);
}
SecretKeySpec key = null;
try {
String algOnly = alg.split("/")[0];
key = new SecretKeySpec(
factory.generateSecret(
new PBEKeySpec(passPhrase.toCharArray(), salt.getBytes(), iterations, keyLen)).getEncoded(),
algOnly);
} catch (Exception e) {
throw new IOException("Can't generate secret key", e);
}
Cipher crypto = null;
try {
crypto = Cipher.getInstance(alg);
} catch (Exception e) {
throw new IOException("Can't initialize the decryptor", e);
}
Path inputFileLocation = Paths.get(inputFileName);
byte[] decrypted = Files.readAllBytes(inputFileLocation);
byte[] encrypted;
try {
crypto.init(Cipher.ENCRYPT_MODE, key);
encrypted = crypto.doFinal(decrypted);
} catch (Exception e) {
throw new IOException("Can't decrypt the password", e);
}
Path outputFileLocation = Paths.get(outputFileName);
Files.write(outputFileLocation, encrypted);
}
}
正如 Simone 的回答 - openssl 和 java 实现之间的加密算法有所不同。这就是为什么您可以毫无问题地使用 openssl 进行解密（因为它再次调用自己的（不同的）算法）。
经过大量挖掘，我从 (Dave Thompson) 中找到了这个答案，其中指出：
Short answer: what openssl enc (without -K for raw) uses is not PBKDF2; it is almost PBKDF1, with iteration count 1.
似乎有两种方法可以解决这个问题，要么：
a) 在 java 中找到可以解密 openssl 正在做什么的东西 - 在这个 [=12] 中的 post 中引用了一个 java 库 'BouncyCastle' =]（如果您喜欢使用它而不是标准的 CryptoFile），他们实现了与 openssl 使用的算法完全相同的算法。
或
b) 找到一些其他的命令行实用程序来代替实现 PBKDF2 的 openssl。 nabble.com posting 中还提到了一些不同语言的实现。
（由于引用了 Dave 的关键观察结果）