Active Directory - Hortonworks 跨领域信任建立
Active Directory - Hortonworks cross realm trust establishment
我们正在尝试为 SAS Data Loader for Hadoop (DLH) 配置 IWA。 SAS 服务器在 Active Directory 域下 运行，并且 SSO 已成功配置。我们需要配置 DLH 以使用客户端生成的票证与 Hortonworks Hadoop MIT Kerberos 对话。该功能不起作用。
所以基本上我们在 AD (ABC.COM) 和 Hadoop MIT Kerberos（xyz - Hadoop 域名没有任何 FQDN 并且全是小写字母）2 路信任建立方面存在问题。我们已经按照以下 link (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/kerb-config-realm-kdc.html) 确定了信任，并且一切正常，但不知何故使用 AD 的 HTTP 票证，我们无法登录到 hadoop，我们收到以下错误消息：
com.sas.svcs.dm.hadoop.spi.exception.HadoopConfigurationException: Failed to find GSSCredential. Check Kerberos configuration
我们已经尝试了很多故障排除方法。现在终于确定我们有信任问题并测试以下是要求执行的步骤。
在 SAS 服务器上 (linux)
kinit -f HTTP/xxx.abc.com@ABC.COM
klist -eaf
kvno hive/xyz@xyz
如果上述所有步骤都有效，则意味着我们已启用信任
这是我们看到的错误
kvno: KDC returned error string: PROCESS_TGS while getting credentials for hive/xyz@xyz
和
kvno: Decrypt integrity check failed while getting credentials for hive/xyz@xyz
kinit -f HTTP/xxx.abc.com@ABC.COM (this works fine)
# kinit -k -t xxx.host.keytab HTTP/xxx.abc.com@ABC.COM
[65181] 1559895039.846538: Getting initial credentials for HTTP/xxx.abc.com@ABC.COM
[65181] 1559895039.846539: Looked up etypes in keytab: des-cbc-crc, des, des-cbc-crc, rc4-hmac, aes256-cts, aes128-cts
[65181] 1559895039.846541: Sending unauthenticated request
[65181] 1559895039.846542: Sending request (220 bytes) to ABC.COM
[65181] 1559895039.846543: Sending initial UDP request to dgram 10.68.5.219:88
[65181] 1559895039.846544: Received answer (819 bytes) from dgram 10.68.5.219:88
[65181] 1559895039.846545: Response was from master KDC
[65181] 1559895039.846546: Processing preauth types: PA-ETYPE-INFO2 (19)
[65181] 1559895039.846547: Selected etype info: etype aes256-cts, salt "ABC.COMHTTPxxx.abc.com", params ""
[65181] 1559895039.846548: Produced preauth for next request: (empty)
[65181] 1559895039.846549: Getting AS key, salt "ABC.COMHTTPxxx.abc.com", params ""
[65181] 1559895039.846550: Retrieving HTTP/xxx.abc.com@ABC.COM from FILE:xxx.host.keytab (vno 0, enctype aes256-cts) with result: 0/Success
[65181] 1559895039.846551: AS key obtained from gak_fct: aes256-cts/8AEB
[65181] 1559895039.846552: Decrypted AS reply; session key is: aes256-cts/E734
[65181] 1559895039.846553: FAST negotiation: unavailable
[65181] 1559895039.846554: Initializing FILE:/tmp/krb5cc_0 with default princ HTTP/xxx.abc.com@ABC.COM
[65181] 1559895039.846555: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM in FILE:/tmp/krb5cc_0
##########################################################
klist -e (this shows the ticket is generated)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/xxx.abc.com@ABC.COM
Valid starting Expires Service principal
06/07/2019 13:40:39 06/07/2019 13:50:39 krbtgt/ABC.COM@ABC.COM
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
###########################################################
kvno hive/xyz@xyz (this command fails)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# kvno hive/xyz@xyz
[65247] 1559895064.242178: Getting credentials HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz using ccache FILE:/tmp/krb5cc_0
[65247] 1559895064.242179: Retrieving HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[65247] 1559895064.242180: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[65247] 1559895064.242181: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[65247] 1559895064.242182: Starting with TGT for client realm: HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM
[65247] 1559895064.242183: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[65247] 1559895064.242184: Requesting TGT krbtgt/xyz@ABC.COM using TGT krbtgt/ABC.COM@ABC.COM
[65247] 1559895064.242185: Generated subkey for TGS request: aes256-cts/C142
[65247] 1559895064.242186: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[65247] 1559895064.242188: Encoding request body and padata into FAST request
[65247] 1559895064.242189: Sending request (1001 bytes) to ABC.COM
[65247] 1559895064.242190: Sending initial UDP request to dgram 10.68.5.219:88
[65247] 1559895064.242191: Received answer (873 bytes) from dgram 10.68.5.219:88
[65247] 1559895064.242192: Response was from master KDC
[65247] 1559895064.242193: Decoding FAST response
[65247] 1559895064.242194: FAST reply key: aes256-cts/9192
[65247] 1559895064.242195: TGS reply is for HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM with session key des-cbc-crc/330F
[65247] 1559895064.242196: TGS request result: 0/Success
[65247] 1559895064.242197: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM in FILE:/tmp/krb5cc_0
[65247] 1559895064.242198: Received TGT for service realm: krbtgt/xyz@ABC.COM
[65247] 1559895064.242199: Requesting tickets for hive/xyz@xyz, referrals on
[65247] 1559895064.242200: Generated subkey for TGS request: des-cbc-crc/FB8F
[65247] 1559895064.242201: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[65247] 1559895064.242203: Encoding request body and padata into FAST request
[65247] 1559895064.242204: Sending request (935 bytes) to xyz
[65247] 1559895064.242205: Resolving hostname xyz
[65247] 1559895064.242206: Sending initial UDP request to dgram 10.68.166.7:88
[65247] 1559895064.242207: Received answer (138 bytes) from dgram 10.68.166.7:88
[65247] 1559895064.242208: Response was not from master KDC
[65247] 1559895064.242209: TGS request result: -1765328324/KDC returned error string: PROCESS_TGS
[65247] 1559895064.242210: Requesting tickets for hive/xyz@xyz, referrals off
[65247] 1559895064.242211: Generated subkey for TGS request: des-cbc-crc/01C2
[65247] 1559895064.242212: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[65247] 1559895064.242214: Encoding request body and padata into FAST request
[65247] 1559895064.242215: Sending request (935 bytes) to xyz
[65247] 1559895064.242216: Resolving hostname xyz
[65247] 1559895064.242217: Sending initial UDP request to dgram 10.68.166.7:88
[65247] 1559895064.242218: Received answer (138 bytes) from dgram 10.68.166.7:88
[65247] 1559895064.242219: Response was not from master KDC
[65247] 1559895064.242220: TGS request result: -1765328324/KDC returned error string: PROCESS_TGS
kvno: KDC returned error string: PROCESS_TGS while getting credentials for hive/xyz@xyz
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Also just for troubleshooting I add enctypes on my AD server using the following command:
ksetup /SetEncTypeAttr xyz DES-CBC-CRC DES-CBC-MD5 RC4-HMAC-MD5 AES128-CTS-HMAC-SHA1-96 AES256-CTS-HMAC-SHA1-96
So, after running the above command when I try to run the kvno command, my error message changes
:from
kvno: KDC returned error string: PROCESS_TGS while getting credentials for hive/xyz@xyz
:to
kvno: Decrypt integrity check failed while getting credentials for hive/xyz@xyz
full kvno cmmand trace is as below:
# kvno hive/xyz@xyz
[128763] 1559917554.849763: Getting credentials HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz using ccache FILE:/tmp/krb5cc_0
[128763] 1559917554.849764: Retrieving HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[128763] 1559917554.849765: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[128763] 1559917554.849766: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[128763] 1559917554.849767: Starting with TGT for client realm: HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM
[128763] 1559917554.849768: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[128763] 1559917554.849769: Requesting TGT krbtgt/xyz@ABC.COM using TGT krbtgt/ABC.COM@ABC.COM
[128763] 1559917554.849770: Generated subkey for TGS request: aes256-cts/4F0F
[128763] 1559917554.849771: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[128763] 1559917554.849773: Encoding request body and padata into FAST request
[128763] 1559917554.849774: Sending request (1022 bytes) to ABC.COM
[128763] 1559917554.849775: Sending initial UDP request to dgram 10.68.5.219:88
[128763] 1559917554.849776: Received answer (969 bytes) from dgram 10.68.5.219:88
[128763] 1559917554.849777: Response was from master KDC
[128763] 1559917554.849778: Decoding FAST response
[128763] 1559917554.849779: FAST reply key: aes256-cts/944C
[128763] 1559917554.849780: TGS reply is for HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM with session key aes256-cts/B3D3
[128763] 1559917554.849781: TGS request result: 0/Success
[128763] 1559917554.849782: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM in FILE:/tmp/krb5cc_0
[128763] 1559917554.849783: Received TGT for service realm: krbtgt/xyz@ABC.COM
[128763] 1559917554.849784: Requesting tickets for hive/xyz@xyz, referrals on
[128763] 1559917554.849785: Generated subkey for TGS request: aes256-cts/DF91
[128763] 1559917554.849786: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[128763] 1559917554.849788: Encoding request body and padata into FAST request
[128763] 1559917554.849789: Sending request (1013 bytes) to xyz
[128763] 1559917554.849790: Resolving hostname xyz
[128763] 1559917554.849791: Sending initial UDP request to dgram 10.68.166.7:88
[128763] 1559917554.849792: Received answer (138 bytes) from dgram 10.68.166.7:88
[128763] 1559917554.849793: Response was not from master KDC
[128763] 1559917554.849794: TGS request result: -1765328353/Decrypt integrity check failed
[128763] 1559917554.849795: Requesting tickets for hive/xyz@xyz, referrals off
[128763] 1559917554.849796: Generated subkey for TGS request: aes256-cts/34D1
[128763] 1559917554.849797: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[128763] 1559917554.849799: Encoding request body and padata into FAST request
[128763] 1559917554.849800: Sending request (1013 bytes) to xyz
[128763] 1559917554.849801: Resolving hostname xyz
[128763] 1559917554.849802: Sending initial UDP request to dgram 10.68.166.7:88
[128763] 1559917554.849803: Received answer (138 bytes) from dgram 10.68.166.7:88
[128763] 1559917554.849805: TGS request result: -1765328353/Decrypt integrity check failed
kvno: Decrypt integrity check failed while getting credentials for hive/xyz@xyz
问题出在 AD 上，Hadoop Trust 工作不正常。因此，在故障排除期间，我在 AD 上的 Hadoop Principal 上添加了 enctype。我在其中一个网站上发现了以下注释
“主体（帐户）是使用 system-default enctype 创建的。当您更改 enctype 时，您还必须重新创建主体，或者至少更新主体的密码。”
所以，我重置了密码
netdom trust xyz /Domain:ABC.COM /reset /realm /passwordt:xxxxXXXxxxx
另外，AD和Hadoop之间的KVNO没有匹配，所以我更新了Hadoop端的kvno
在 Hadoop 服务器上重新启动了以下服务
/sbin/service krb5kdc 重启
/sbin/service kadmin 重启
瞧瞧...我能够 运行 kvno 命令。
kinit -k -t xxx.host.keytab HTTP/xxx.abc.com@ABC.COM
[74264] 1561019777.500742：存储 HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM 在文件中：/tmp/krb5cc_1001
klist -eaf
票证缓存：文件：/tmp/krb5cc_1001
默认主体：HTTP/xxx.abc.com@ABC.COM
有效的起始 Expires 服务主体
06/20/2019 14:06:17 06/21/2019 00:06:17 krbtgt/ABC.COM@ABC.COM
续订至 06/27/2019 14:06:17，标记：周五
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
地址：(none)
kvno hive/xyz@xyz
[74362] 1561019789.592571：收到所需服务的信用 hive/xyz@xyz
[74362] 1561019789.592572：存储 HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz 在文件中：/tmp/krb5cc_1001
hive/xyz@xyz: kvno = 1
我们正在尝试为 SAS Data Loader for Hadoop (DLH) 配置 IWA。 SAS 服务器在 Active Directory 域下 运行，并且 SSO 已成功配置。我们需要配置 DLH 以使用客户端生成的票证与 Hortonworks Hadoop MIT Kerberos 对话。该功能不起作用。
所以基本上我们在 AD (ABC.COM) 和 Hadoop MIT Kerberos（xyz - Hadoop 域名没有任何 FQDN 并且全是小写字母）2 路信任建立方面存在问题。我们已经按照以下 link (https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.6.0/bk_security/content/kerb-config-realm-kdc.html) 确定了信任，并且一切正常，但不知何故使用 AD 的 HTTP 票证，我们无法登录到 hadoop，我们收到以下错误消息：
com.sas.svcs.dm.hadoop.spi.exception.HadoopConfigurationException: Failed to find GSSCredential. Check Kerberos configuration
我们已经尝试了很多故障排除方法。现在终于确定我们有信任问题并测试以下是要求执行的步骤。
在 SAS 服务器上 (linux)
kinit -f HTTP/xxx.abc.com@ABC.COM
klist -eaf
kvno hive/xyz@xyz
如果上述所有步骤都有效，则意味着我们已启用信任
这是我们看到的错误
kvno: KDC returned error string: PROCESS_TGS while getting credentials for hive/xyz@xyz
和
kvno: Decrypt integrity check failed while getting credentials for hive/xyz@xyz
kinit -f HTTP/xxx.abc.com@ABC.COM (this works fine)
# kinit -k -t xxx.host.keytab HTTP/xxx.abc.com@ABC.COM
[65181] 1559895039.846538: Getting initial credentials for HTTP/xxx.abc.com@ABC.COM
[65181] 1559895039.846539: Looked up etypes in keytab: des-cbc-crc, des, des-cbc-crc, rc4-hmac, aes256-cts, aes128-cts
[65181] 1559895039.846541: Sending unauthenticated request
[65181] 1559895039.846542: Sending request (220 bytes) to ABC.COM
[65181] 1559895039.846543: Sending initial UDP request to dgram 10.68.5.219:88
[65181] 1559895039.846544: Received answer (819 bytes) from dgram 10.68.5.219:88
[65181] 1559895039.846545: Response was from master KDC
[65181] 1559895039.846546: Processing preauth types: PA-ETYPE-INFO2 (19)
[65181] 1559895039.846547: Selected etype info: etype aes256-cts, salt "ABC.COMHTTPxxx.abc.com", params ""
[65181] 1559895039.846548: Produced preauth for next request: (empty)
[65181] 1559895039.846549: Getting AS key, salt "ABC.COMHTTPxxx.abc.com", params ""
[65181] 1559895039.846550: Retrieving HTTP/xxx.abc.com@ABC.COM from FILE:xxx.host.keytab (vno 0, enctype aes256-cts) with result: 0/Success
[65181] 1559895039.846551: AS key obtained from gak_fct: aes256-cts/8AEB
[65181] 1559895039.846552: Decrypted AS reply; session key is: aes256-cts/E734
[65181] 1559895039.846553: FAST negotiation: unavailable
[65181] 1559895039.846554: Initializing FILE:/tmp/krb5cc_0 with default princ HTTP/xxx.abc.com@ABC.COM
[65181] 1559895039.846555: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM in FILE:/tmp/krb5cc_0
##########################################################
klist -e (this shows the ticket is generated)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# klist -e
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: HTTP/xxx.abc.com@ABC.COM
Valid starting Expires Service principal
06/07/2019 13:40:39 06/07/2019 13:50:39 krbtgt/ABC.COM@ABC.COM
Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96
###########################################################
kvno hive/xyz@xyz (this command fails)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# kvno hive/xyz@xyz
[65247] 1559895064.242178: Getting credentials HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz using ccache FILE:/tmp/krb5cc_0
[65247] 1559895064.242179: Retrieving HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[65247] 1559895064.242180: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[65247] 1559895064.242181: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[65247] 1559895064.242182: Starting with TGT for client realm: HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM
[65247] 1559895064.242183: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[65247] 1559895064.242184: Requesting TGT krbtgt/xyz@ABC.COM using TGT krbtgt/ABC.COM@ABC.COM
[65247] 1559895064.242185: Generated subkey for TGS request: aes256-cts/C142
[65247] 1559895064.242186: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[65247] 1559895064.242188: Encoding request body and padata into FAST request
[65247] 1559895064.242189: Sending request (1001 bytes) to ABC.COM
[65247] 1559895064.242190: Sending initial UDP request to dgram 10.68.5.219:88
[65247] 1559895064.242191: Received answer (873 bytes) from dgram 10.68.5.219:88
[65247] 1559895064.242192: Response was from master KDC
[65247] 1559895064.242193: Decoding FAST response
[65247] 1559895064.242194: FAST reply key: aes256-cts/9192
[65247] 1559895064.242195: TGS reply is for HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM with session key des-cbc-crc/330F
[65247] 1559895064.242196: TGS request result: 0/Success
[65247] 1559895064.242197: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM in FILE:/tmp/krb5cc_0
[65247] 1559895064.242198: Received TGT for service realm: krbtgt/xyz@ABC.COM
[65247] 1559895064.242199: Requesting tickets for hive/xyz@xyz, referrals on
[65247] 1559895064.242200: Generated subkey for TGS request: des-cbc-crc/FB8F
[65247] 1559895064.242201: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[65247] 1559895064.242203: Encoding request body and padata into FAST request
[65247] 1559895064.242204: Sending request (935 bytes) to xyz
[65247] 1559895064.242205: Resolving hostname xyz
[65247] 1559895064.242206: Sending initial UDP request to dgram 10.68.166.7:88
[65247] 1559895064.242207: Received answer (138 bytes) from dgram 10.68.166.7:88
[65247] 1559895064.242208: Response was not from master KDC
[65247] 1559895064.242209: TGS request result: -1765328324/KDC returned error string: PROCESS_TGS
[65247] 1559895064.242210: Requesting tickets for hive/xyz@xyz, referrals off
[65247] 1559895064.242211: Generated subkey for TGS request: des-cbc-crc/01C2
[65247] 1559895064.242212: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[65247] 1559895064.242214: Encoding request body and padata into FAST request
[65247] 1559895064.242215: Sending request (935 bytes) to xyz
[65247] 1559895064.242216: Resolving hostname xyz
[65247] 1559895064.242217: Sending initial UDP request to dgram 10.68.166.7:88
[65247] 1559895064.242218: Received answer (138 bytes) from dgram 10.68.166.7:88
[65247] 1559895064.242219: Response was not from master KDC
[65247] 1559895064.242220: TGS request result: -1765328324/KDC returned error string: PROCESS_TGS
kvno: KDC returned error string: PROCESS_TGS while getting credentials for hive/xyz@xyz
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
Also just for troubleshooting I add enctypes on my AD server using the following command:
ksetup /SetEncTypeAttr xyz DES-CBC-CRC DES-CBC-MD5 RC4-HMAC-MD5 AES128-CTS-HMAC-SHA1-96 AES256-CTS-HMAC-SHA1-96
So, after running the above command when I try to run the kvno command, my error message changes
:from
kvno: KDC returned error string: PROCESS_TGS while getting credentials for hive/xyz@xyz
:to
kvno: Decrypt integrity check failed while getting credentials for hive/xyz@xyz
full kvno cmmand trace is as below:
# kvno hive/xyz@xyz
[128763] 1559917554.849763: Getting credentials HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz using ccache FILE:/tmp/krb5cc_0
[128763] 1559917554.849764: Retrieving HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[128763] 1559917554.849765: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[128763] 1559917554.849766: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM from FILE:/tmp/krb5cc_0 with result: 0/Success
[128763] 1559917554.849767: Starting with TGT for client realm: HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM
[128763] 1559917554.849768: Retrieving HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@xyz from FILE:/tmp/krb5cc_0 with result: -1765328243/Matching credential not found (filename: /tmp/krb5cc_0)
[128763] 1559917554.849769: Requesting TGT krbtgt/xyz@ABC.COM using TGT krbtgt/ABC.COM@ABC.COM
[128763] 1559917554.849770: Generated subkey for TGS request: aes256-cts/4F0F
[128763] 1559917554.849771: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[128763] 1559917554.849773: Encoding request body and padata into FAST request
[128763] 1559917554.849774: Sending request (1022 bytes) to ABC.COM
[128763] 1559917554.849775: Sending initial UDP request to dgram 10.68.5.219:88
[128763] 1559917554.849776: Received answer (969 bytes) from dgram 10.68.5.219:88
[128763] 1559917554.849777: Response was from master KDC
[128763] 1559917554.849778: Decoding FAST response
[128763] 1559917554.849779: FAST reply key: aes256-cts/944C
[128763] 1559917554.849780: TGS reply is for HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM with session key aes256-cts/B3D3
[128763] 1559917554.849781: TGS request result: 0/Success
[128763] 1559917554.849782: Storing HTTP/xxx.abc.com@ABC.COM -> krbtgt/xyz@ABC.COM in FILE:/tmp/krb5cc_0
[128763] 1559917554.849783: Received TGT for service realm: krbtgt/xyz@ABC.COM
[128763] 1559917554.849784: Requesting tickets for hive/xyz@xyz, referrals on
[128763] 1559917554.849785: Generated subkey for TGS request: aes256-cts/DF91
[128763] 1559917554.849786: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[128763] 1559917554.849788: Encoding request body and padata into FAST request
[128763] 1559917554.849789: Sending request (1013 bytes) to xyz
[128763] 1559917554.849790: Resolving hostname xyz
[128763] 1559917554.849791: Sending initial UDP request to dgram 10.68.166.7:88
[128763] 1559917554.849792: Received answer (138 bytes) from dgram 10.68.166.7:88
[128763] 1559917554.849793: Response was not from master KDC
[128763] 1559917554.849794: TGS request result: -1765328353/Decrypt integrity check failed
[128763] 1559917554.849795: Requesting tickets for hive/xyz@xyz, referrals off
[128763] 1559917554.849796: Generated subkey for TGS request: aes256-cts/34D1
[128763] 1559917554.849797: etypes requested in TGS request: aes256-cts, aes128-cts, aes256-sha2, aes128-sha2, des3-cbc-sha1, rc4-hmac, camellia128-cts, camellia256-cts, des-cbc-crc, des, des-cbc-md4
[128763] 1559917554.849799: Encoding request body and padata into FAST request
[128763] 1559917554.849800: Sending request (1013 bytes) to xyz
[128763] 1559917554.849801: Resolving hostname xyz
[128763] 1559917554.849802: Sending initial UDP request to dgram 10.68.166.7:88
[128763] 1559917554.849803: Received answer (138 bytes) from dgram 10.68.166.7:88
[128763] 1559917554.849805: TGS request result: -1765328353/Decrypt integrity check failed
kvno: Decrypt integrity check failed while getting credentials for hive/xyz@xyz
问题出在 AD 上，Hadoop Trust 工作不正常。因此，在故障排除期间，我在 AD 上的 Hadoop Principal 上添加了 enctype。我在其中一个网站上发现了以下注释
“主体（帐户）是使用 system-default enctype 创建的。当您更改 enctype 时，您还必须重新创建主体，或者至少更新主体的密码。”
所以，我重置了密码
netdom trust xyz /Domain:ABC.COM /reset /realm /passwordt:xxxxXXXxxxx
另外，AD和Hadoop之间的KVNO没有匹配，所以我更新了Hadoop端的kvno
在 Hadoop 服务器上重新启动了以下服务
/sbin/service krb5kdc 重启 /sbin/service kadmin 重启
瞧瞧...我能够 运行 kvno 命令。
kinit -k -t xxx.host.keytab HTTP/xxx.abc.com@ABC.COM
[74264] 1561019777.500742：存储 HTTP/xxx.abc.com@ABC.COM -> krbtgt/ABC.COM@ABC.COM 在文件中：/tmp/krb5cc_1001
klist -eaf
票证缓存：文件：/tmp/krb5cc_1001 默认主体：HTTP/xxx.abc.com@ABC.COM
有效的起始 Expires 服务主体 06/20/2019 14:06:17 06/21/2019 00:06:17 krbtgt/ABC.COM@ABC.COM 续订至 06/27/2019 14:06:17，标记：周五 Etype (skey, tkt): aes256-cts-hmac-sha1-96, aes256-cts-hmac-sha1-96 地址：(none)
kvno hive/xyz@xyz
[74362] 1561019789.592571：收到所需服务的信用 hive/xyz@xyz [74362] 1561019789.592572：存储 HTTP/xxx.abc.com@ABC.COM -> hive/xyz@xyz 在文件中：/tmp/krb5cc_1001 hive/xyz@xyz: kvno = 1