提前终止 SSL
Early SSL Termination
SSL 在 VPN 客户端上工作，但在 GCP-PaloAlto VPN 隧道上失败。
服务器是本地 Keycloak（版本 13.0.0）
从 VPN 客户端尝试（成功）
# openssl s_client -connect fqdn:443 -servername fqdn -tls1_2 --prexit
CONNECTED(00000005)
[...]
depth=0 CN = <fqdn>
verify return:1
---
Certificate chain
0 s:CN = <fqdn>
[...]
---
Server certificate
[...]
subject=CN = <fqdn>
issuer=C = US, O = Let's Encrypt, CN = R3
---
[...]
---
SSL handshake has read 4672 bytes and written 311 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
[...]
---
closed
---
Certificate chain
0 s:CN = <fqdn>
[...]
---
Server certificate
[...]
subject=CN = <fqdn>
issuer=C = US, O = Let's Encrypt, CN = R3
[...]
来自工作客户端的 tcpdump
19:32:09.582389 IP CLIENT.51013 > SERVER.https: Flags [S], seq 3647907685, win 65535, options [mss 1360,nop,wscale 6,nop,nop,TS val 350978343 ecr 0,sackOK,eol], length 0
19:32:09.582551 IP SERVER.https > CLIENT.51013: Flags [S.], seq 3495009322, ack 3647907686, win 65160, options [mss 1460,sackOK,TS val 2099619205 ecr 350978343,nop,wscale 7], length 0
19:32:09.596385 IP CLIENT.51013 > SERVER.https: Flags [.], ack 1, win 2064, options [nop,nop,TS val 350978356 ecr 2099619205], length 0
19:32:09.596385 IP CLIENT.51013 > SERVER.https: Flags [P.], seq 1:219, ack 1, win 2064, options [nop,nop,TS val 350978356 ecr 2099619205], length 218
19:32:09.596502 IP SERVER.https > CLIENT.51013: Flags [.], ack 219, win 508, options [nop,nop,TS val 2099619219 ecr 350978356], length 0
19:32:09.596976 IP SERVER.https > CLIENT.51013: Flags [P.], seq 1:2697, ack 219, win 508, options [nop,nop,TS val 2099619219 ecr 350978356], length 2696
19:32:09.597009 IP SERVER.https > CLIENT.51013: Flags [P.], seq 2697:4097, ack 219, win 508, options [nop,nop,TS val 2099619219 ecr 350978356], length 1400
19:32:09.599161 IP SERVER.https > CLIENT.51013: Flags [P.], seq 4097:4415, ack 219, win 508, options [nop,nop,TS val 2099619221 ecr 350978356], length 318
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 1349, win 2043, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 2697, win 2022, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4045, win 2000, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4097, win 2000, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4097, win 2048, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.614194 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4415, win 2043, options [nop,nop,TS val 350978369 ecr 2099619221], length 0
19:32:09.614194 IP CLIENT.51013 > SERVER.https: Flags [P.], seq 219:312, ack 4415, win 2048, options [nop,nop,TS val 350978370 ecr 2099619221], length 93
19:32:09.614248 IP SERVER.https > CLIENT.51013: Flags [.], ack 312, win 508, options [nop,nop,TS val 2099619236 ecr 350978370], length 0
19:32:09.614711 IP SERVER.https > CLIENT.51013: Flags [P.], seq 4415:4673, ack 312, win 508, options [nop,nop,TS val 2099619237 ecr 350978370], length 258
19:32:09.626178 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4673, win 2043, options [nop,nop,TS val 350978384 ecr 2099619237], length 0
19:33:09.651599 IP SERVER.https > CLIENT.51013: Flags [P.], seq 4673:4704, ack 312, win 508, options [nop,nop,TS val 2099679274 ecr 350978384], length 31
19:33:09.651690 IP SERVER.https > CLIENT.51013: Flags [F.], seq 4704, ack 312, win 508, options [nop,nop,TS val 2099679274 ecr 350978384], length 0
19:33:09.678658 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4704, win 2047, options [nop,nop,TS val 351038150 ecr 2099679274], length 0
19:33:09.678806 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4705, win 2047, options [nop,nop,TS val 351038150 ecr 2099679274], length 0
19:33:09.680749 IP CLIENT.51013 > SERVER.https: Flags [P.], seq 312:343, ack 4705, win 2048, options [nop,nop,TS val 351038150 ecr 2099679274], length 31
19:33:09.680749 IP CLIENT.51013 > SERVER.https: Flags [F.], seq 343, ack 4705, win 2048, options [nop,nop,TS val 351038150 ecr 2099679274], length 0
19:33:09.680824 IP SERVER.https > CLIENT.51013: Flags [R], seq 3495014027, win 0, length 0
19:33:09.680888 IP SERVER.https > CLIENT.51013: Flags [R], seq 3495014027, win 0, length 0
从 GCP 实例尝试（不成功）
# openssl s_client -connect fqdn:443 -servername fqdn -tls1_2 --prexit
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 212 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
[...]
SSL-Session:
Protocol : TLSv1.2
[...]
---
来自客户端的 tcpdump 不起作用
19:36:01.818130 IP CLIENT.55282 > SERVER.https: Flags [S], seq 3880590205, win 65320, options [mss 1350,sackOK,TS val 3904950778 ecr 0,nop,wscale 7], length 0
19:36:01.818241 IP SERVER.https > CLIENT.55282: Flags [S.], seq 1198318204, ack 3880590206, win 65160, options [mss 1460,sackOK,TS val 2676835275 ecr 3904950778,nop,wscale 7], length 0
19:36:01.829890 IP CLIENT.55282 > SERVER.https: Flags [.], ack 1, win 511, options [nop,nop,TS val 3904950791 ecr 2676835275], length 0
19:37:01.890295 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676895347 ecr 3904950791], length 0
19:37:02.105125 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676895562 ecr 3904950791], length 0
19:37:02.321181 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676895778 ecr 3904950791], length 0
19:37:02.753156 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676896210 ecr 3904950791], length 0
19:37:03.617211 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676897074 ecr 3904950791], length 0
19:37:05.345180 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676898802 ecr 3904950791], length 0
19:37:08.769186 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676902226 ecr 3904950791], length 0
19:37:15.681150 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676909138 ecr 3904950791], length 0
不确定这是否重要，但我观察到的差异是网络的 MTU。
VPN 客户端最多可以 ping 1372 大小的服务器
GCP 节点最多可以 ping 1362 大小的服务器
原来是异步路由的情况。两个本地站点通告相同的路由，但其中只有一个有网络。
总结：
SYN: GCP -> SITE A
ACK, SYN: SITE A -> GCP
ACK: GCP -(through SITE B)-> SITE A
FIN: SITE A doesn't like that the ACK came though a different tunnel then the one used to send the SYN.
Terminates the connection and sends a FIN.
SSL 在 VPN 客户端上工作，但在 GCP-PaloAlto VPN 隧道上失败。
服务器是本地 Keycloak（版本 13.0.0）
从 VPN 客户端尝试（成功）
# openssl s_client -connect fqdn:443 -servername fqdn -tls1_2 --prexit
CONNECTED(00000005)
[...]
depth=0 CN = <fqdn>
verify return:1
---
Certificate chain
0 s:CN = <fqdn>
[...]
---
Server certificate
[...]
subject=CN = <fqdn>
issuer=C = US, O = Let's Encrypt, CN = R3
---
[...]
---
SSL handshake has read 4672 bytes and written 311 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
[...]
---
closed
---
Certificate chain
0 s:CN = <fqdn>
[...]
---
Server certificate
[...]
subject=CN = <fqdn>
issuer=C = US, O = Let's Encrypt, CN = R3
[...]
来自工作客户端的 tcpdump
19:32:09.582389 IP CLIENT.51013 > SERVER.https: Flags [S], seq 3647907685, win 65535, options [mss 1360,nop,wscale 6,nop,nop,TS val 350978343 ecr 0,sackOK,eol], length 0
19:32:09.582551 IP SERVER.https > CLIENT.51013: Flags [S.], seq 3495009322, ack 3647907686, win 65160, options [mss 1460,sackOK,TS val 2099619205 ecr 350978343,nop,wscale 7], length 0
19:32:09.596385 IP CLIENT.51013 > SERVER.https: Flags [.], ack 1, win 2064, options [nop,nop,TS val 350978356 ecr 2099619205], length 0
19:32:09.596385 IP CLIENT.51013 > SERVER.https: Flags [P.], seq 1:219, ack 1, win 2064, options [nop,nop,TS val 350978356 ecr 2099619205], length 218
19:32:09.596502 IP SERVER.https > CLIENT.51013: Flags [.], ack 219, win 508, options [nop,nop,TS val 2099619219 ecr 350978356], length 0
19:32:09.596976 IP SERVER.https > CLIENT.51013: Flags [P.], seq 1:2697, ack 219, win 508, options [nop,nop,TS val 2099619219 ecr 350978356], length 2696
19:32:09.597009 IP SERVER.https > CLIENT.51013: Flags [P.], seq 2697:4097, ack 219, win 508, options [nop,nop,TS val 2099619219 ecr 350978356], length 1400
19:32:09.599161 IP SERVER.https > CLIENT.51013: Flags [P.], seq 4097:4415, ack 219, win 508, options [nop,nop,TS val 2099619221 ecr 350978356], length 318
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 1349, win 2043, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 2697, win 2022, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4045, win 2000, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4097, win 2000, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.612262 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4097, win 2048, options [nop,nop,TS val 350978368 ecr 2099619219], length 0
19:32:09.614194 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4415, win 2043, options [nop,nop,TS val 350978369 ecr 2099619221], length 0
19:32:09.614194 IP CLIENT.51013 > SERVER.https: Flags [P.], seq 219:312, ack 4415, win 2048, options [nop,nop,TS val 350978370 ecr 2099619221], length 93
19:32:09.614248 IP SERVER.https > CLIENT.51013: Flags [.], ack 312, win 508, options [nop,nop,TS val 2099619236 ecr 350978370], length 0
19:32:09.614711 IP SERVER.https > CLIENT.51013: Flags [P.], seq 4415:4673, ack 312, win 508, options [nop,nop,TS val 2099619237 ecr 350978370], length 258
19:32:09.626178 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4673, win 2043, options [nop,nop,TS val 350978384 ecr 2099619237], length 0
19:33:09.651599 IP SERVER.https > CLIENT.51013: Flags [P.], seq 4673:4704, ack 312, win 508, options [nop,nop,TS val 2099679274 ecr 350978384], length 31
19:33:09.651690 IP SERVER.https > CLIENT.51013: Flags [F.], seq 4704, ack 312, win 508, options [nop,nop,TS val 2099679274 ecr 350978384], length 0
19:33:09.678658 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4704, win 2047, options [nop,nop,TS val 351038150 ecr 2099679274], length 0
19:33:09.678806 IP CLIENT.51013 > SERVER.https: Flags [.], ack 4705, win 2047, options [nop,nop,TS val 351038150 ecr 2099679274], length 0
19:33:09.680749 IP CLIENT.51013 > SERVER.https: Flags [P.], seq 312:343, ack 4705, win 2048, options [nop,nop,TS val 351038150 ecr 2099679274], length 31
19:33:09.680749 IP CLIENT.51013 > SERVER.https: Flags [F.], seq 343, ack 4705, win 2048, options [nop,nop,TS val 351038150 ecr 2099679274], length 0
19:33:09.680824 IP SERVER.https > CLIENT.51013: Flags [R], seq 3495014027, win 0, length 0
19:33:09.680888 IP SERVER.https > CLIENT.51013: Flags [R], seq 3495014027, win 0, length 0
从 GCP 实例尝试（不成功）
# openssl s_client -connect fqdn:443 -servername fqdn -tls1_2 --prexit
CONNECTED(00000003)
write:errno=0
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 212 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
[...]
SSL-Session:
Protocol : TLSv1.2
[...]
---
来自客户端的 tcpdump 不起作用
19:36:01.818130 IP CLIENT.55282 > SERVER.https: Flags [S], seq 3880590205, win 65320, options [mss 1350,sackOK,TS val 3904950778 ecr 0,nop,wscale 7], length 0
19:36:01.818241 IP SERVER.https > CLIENT.55282: Flags [S.], seq 1198318204, ack 3880590206, win 65160, options [mss 1460,sackOK,TS val 2676835275 ecr 3904950778,nop,wscale 7], length 0
19:36:01.829890 IP CLIENT.55282 > SERVER.https: Flags [.], ack 1, win 511, options [nop,nop,TS val 3904950791 ecr 2676835275], length 0
19:37:01.890295 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676895347 ecr 3904950791], length 0
19:37:02.105125 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676895562 ecr 3904950791], length 0
19:37:02.321181 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676895778 ecr 3904950791], length 0
19:37:02.753156 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676896210 ecr 3904950791], length 0
19:37:03.617211 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676897074 ecr 3904950791], length 0
19:37:05.345180 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676898802 ecr 3904950791], length 0
19:37:08.769186 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676902226 ecr 3904950791], length 0
19:37:15.681150 IP SERVER.https > CLIENT.55282: Flags [F.], seq 1, ack 1, win 510, options [nop,nop,TS val 2676909138 ecr 3904950791], length 0
不确定这是否重要，但我观察到的差异是网络的 MTU。
VPN 客户端最多可以 ping 1372 大小的服务器
GCP 节点最多可以 ping 1362 大小的服务器
原来是异步路由的情况。两个本地站点通告相同的路由，但其中只有一个有网络。
总结：
SYN: GCP -> SITE A
ACK, SYN: SITE A -> GCP
ACK: GCP -(through SITE B)-> SITE A
FIN: SITE A doesn't like that the ACK came though a different tunnel then the one used to send the SYN.
Terminates the connection and sends a FIN.