用户的 Kerberized Hadoop 登录失败 ... LoginException:校验和失败
Kerberized Hadoop Login failure for user ... LoginException: Checksum failed
环境:
- Hadoop 2.9.2
- Kerberos 5 版本 1.15.1
- RHEL 7
错误
Hadoop 数据节点日志中的异常阻止启动。
日志条目是:
INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.security.KerberosAuthException: Login failure for user: datanode/_HOST@<REALM> from keytab /etc/security/keytabs/<file.keytab> javax.security.auth.login.LoginException: Checksum failed
完整堆栈跟踪:
org.apache.hadoop.security.KerberosAuthException: Login failure for user: datanode/_HOST@<REALM> from keytab /etc/security/keytabs/datanode.keytab javax.security.auth.login.LoginException: Checksum failed
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1104)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:312)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2596)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2645)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2789)
at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: javax.security.auth.login.LoginException: Checksum failed
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:808)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access[=11=]0(LoginContext.java:195)
at javax.security.auth.login.LoginContext.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1095)
... 10 more
Caused by: KrbException: Checksum failed
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102)
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)
at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:150)
at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:308)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:447)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:780)
... 23 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451)
at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272)
at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76)
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100)
... 30 more
诊断
如果 kdiag 是 运行 诊断问题:
bin/hadoop org.apache.hadoop.security.KDiag --principal namenode/_HOST@<REALM> --keytab /etc/security/keytab/namenode.keytab
遇到和上面一样的异常...
最后一行有用的输出是:
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
正常的健康输出应该是:
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes256CksumType
>>> KrbAsRep cons in KrbAsReq.getReply datanode/_HOST
/etc/krb5.conf
Kerberos 配置的内容
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
rdns = false
forwardable = true
# pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
default_realm = EXAMPLE.COM
default_ccache_name = KEYRING:persistent:%{uid}
[realms]
EXAMPLE.COM = {
kdc = kdc.example.com
admin_server = kdc.example.com
dict_file = /usr/share/dict/words
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
Hadoop 文档解释说这是 Java 不支持可更新票证的问题。
1.8.0_242 Kerberos Java client will fail by "Message stream modified (41)" when the client requests a renewable ticket and the KDC
returns a non-renewable ticket. If your principal is not allowed to
obtain a renewable ticket, you must remove "renew_lifetime" setting
from your krb5.conf.
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
选项 1
从 krb5.conf
中删除 renew_lifetime
注意:如果您不想在系统范围内禁用可再生 kerberos 票证,您可以通过传递这些参数为 java
配置自定义 krb5.conf
jvm 的参数:
# non-windows
-Djava.security.krb5.conf=krb5.conf
# windows
-Djava.security.krb5.conf=krb5.ini
根据this
选项 2
确保 renew_lifetime
、ticket_lifetime
和 max_renewable_life
已设置。
示例工作配置
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
default_tgs_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des- cbc-crc
default_tkt_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc
permitted_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc
kdc_timeout = 3000
[realms]
EXAMPLE.COM = {
kdc = kdc.example.com
admin_server = kdc.example.com
dict_file = /usr/share/dict/words
max_renewable_life = 7d 0h 0m 0s
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
环境:
- Hadoop 2.9.2
- Kerberos 5 版本 1.15.1
- RHEL 7
错误
Hadoop 数据节点日志中的异常阻止启动。
日志条目是:
INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1: org.apache.hadoop.security.KerberosAuthException: Login failure for user: datanode/_HOST@<REALM> from keytab /etc/security/keytabs/<file.keytab> javax.security.auth.login.LoginException: Checksum failed
完整堆栈跟踪:
org.apache.hadoop.security.KerberosAuthException: Login failure for user: datanode/_HOST@<REALM> from keytab /etc/security/keytabs/datanode.keytab javax.security.auth.login.LoginException: Checksum failed
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1104)
at org.apache.hadoop.security.SecurityUtil.login(SecurityUtil.java:312)
at org.apache.hadoop.hdfs.server.datanode.DataNode.instantiateDataNode(DataNode.java:2596)
at org.apache.hadoop.hdfs.server.datanode.DataNode.createDataNode(DataNode.java:2645)
at org.apache.hadoop.hdfs.server.datanode.DataNode.secureMain(DataNode.java:2789)
at org.apache.hadoop.hdfs.server.datanode.SecureDataNodeStarter.start(SecureDataNodeStarter.java:77)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.commons.daemon.support.DaemonLoader.start(DaemonLoader.java:243)
Caused by: javax.security.auth.login.LoginException: Checksum failed
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:808)
at com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
at javax.security.auth.login.LoginContext.access[=11=]0(LoginContext.java:195)
at javax.security.auth.login.LoginContext.run(LoginContext.java:682)
at javax.security.auth.login.LoginContext.run(LoginContext.java:680)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
at org.apache.hadoop.security.UserGroupInformation.loginUserFromKeytab(UserGroupInformation.java:1095)
... 10 more
Caused by: KrbException: Checksum failed
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:102)
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:94)
at sun.security.krb5.EncryptedData.decrypt(EncryptedData.java:175)
at sun.security.krb5.KrbAsRep.decrypt(KrbAsRep.java:150)
at sun.security.krb5.KrbAsRep.decryptUsingKeyTab(KrbAsRep.java:121)
at sun.security.krb5.KrbAsReqBuilder.resolve(KrbAsReqBuilder.java:308)
at sun.security.krb5.KrbAsReqBuilder.action(KrbAsReqBuilder.java:447)
at com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:780)
... 23 more
Caused by: java.security.GeneralSecurityException: Checksum failed
at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decryptCTS(AesDkCrypto.java:451)
at sun.security.krb5.internal.crypto.dk.AesDkCrypto.decrypt(AesDkCrypto.java:272)
at sun.security.krb5.internal.crypto.Aes256.decrypt(Aes256.java:76)
at sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType.decrypt(Aes256CtsHmacSha1EType.java:100)
... 30 more
诊断
如果 kdiag 是 运行 诊断问题:
bin/hadoop org.apache.hadoop.security.KDiag --principal namenode/_HOST@<REALM> --keytab /etc/security/keytab/namenode.keytab
遇到和上面一样的异常...
最后一行有用的输出是:
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
正常的健康输出应该是:
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes256CksumType
>>> KrbAsRep cons in KrbAsReq.getReply datanode/_HOST
/etc/krb5.conf
Kerberos 配置的内容
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
dns_lookup_realm = false
dns_lookup_kdc = false
ticket_lifetime = 24h
renew_lifetime = 7d
rdns = false
forwardable = true
# pkinit_anchors = /etc/pki/tls/certs/ca-bundle.crt
default_realm = EXAMPLE.COM
default_ccache_name = KEYRING:persistent:%{uid}
[realms]
EXAMPLE.COM = {
kdc = kdc.example.com
admin_server = kdc.example.com
dict_file = /usr/share/dict/words
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM
Hadoop 文档解释说这是 Java 不支持可更新票证的问题。
1.8.0_242 Kerberos Java client will fail by "Message stream modified (41)" when the client requests a renewable ticket and the KDC returns a non-renewable ticket. If your principal is not allowed to obtain a renewable ticket, you must remove "renew_lifetime" setting from your krb5.conf.
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+Java+Versions
选项 1
从 krb5.conf
renew_lifetime
注意:如果您不想在系统范围内禁用可再生 kerberos 票证,您可以通过传递这些参数为 java
配置自定义 krb5.conf
jvm 的参数:
# non-windows
-Djava.security.krb5.conf=krb5.conf
# windows
-Djava.security.krb5.conf=krb5.ini
根据this
选项 2
确保 renew_lifetime
、ticket_lifetime
和 max_renewable_life
已设置。
示例工作配置
[logging]
default = FILE:/var/log/krb5libs.log
kdc = FILE:/var/log/krb5kdc.log
admin_server = FILE:/var/log/kadmind.log
[libdefaults]
default_realm = EXAMPLE.COM
dns_lookup_kdc = false
dns_lookup_realm = false
ticket_lifetime = 24h
renew_lifetime = 7d
forwardable = true
default_tgs_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des- cbc-crc
default_tkt_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc
permitted_enctypes = aes256-cts aes128-cts des3-hmac-sha1 arcfour-hmac des-hmac-sha1 des-cbc-md5 des-cbc-crc
kdc_timeout = 3000
[realms]
EXAMPLE.COM = {
kdc = kdc.example.com
admin_server = kdc.example.com
dict_file = /usr/share/dict/words
max_renewable_life = 7d 0h 0m 0s
}
[domain_realm]
.example.com = EXAMPLE.COM
example.com = EXAMPLE.COM