netty 4客户端ssl配置请求所有https站点
netty 4 client ssl configuration to request all https sites
我有一些基于 netty (4.1b7) 的网络爬虫,我在其中大量请求不同的站点,包括 http 和 https 站点,并且我正在尝试配置 netty 客户端以处理具有不同身份验证设置的 https 站点。
当我有一个简单的 netty 配置时 w/o 自己的证书:
SslContext sslCtx = SslContextBuilder.forClient().build();
SSLEngine sslEngine = sslCtx.newEngine(ch.alloc(), host, port);
p.addLast("ssl", new SslHandler(sslEngine));
大约一半的 https 站点被请求成功,但其他站点失败:
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1506)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker.run(Handshaker.java:919)
at sun.security.ssl.Handshaker.run(Handshaker.java:916)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1164)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1067)
... 19 moreCaused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1493)
... 27 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
... 33 more
或:
Caused by: javax.net.ssl.SSLException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1138)
当我尝试 generate own local certificates 并将它们设置为:
System.setProperty("javax.net.ssl.trustStore", "/etc/ssl/my/cacerts.jks");
System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
System.setProperty("javax.net.ssl.keyStore", "/etc/ssl/my/keystore.jks");
System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
然后所有 https 站点都失败并出现如下错误:
Caused by: java.security.cert.CertificateException: found no certificates: /etc/ssl/my/cacerts.jks
at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:83) ~[netty-all-4.1.0.Beta7.jar:4.1.0.Beta7]
at io.netty.handler.ssl.SslContext.toX509Certificates(SslContext.java:967)
....
Caused by: java.security.KeyException: found no private key: /etc/ssl/my/keystore.jks
at io.netty.handler.ssl.PemReader.readPrivateKey(PemReader.java:99) ~[netty-all-4.1.0.Beta7.jar:4.1.0.Beta7]
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:923)
我也尝试了建议 from that SO,但还没有成功。
出了什么问题或者任何人都可以提供某种逐步指南来配置 netty 4+ 客户端以处理具有所有可能的身份验证设置的 https 站点。
尝试像下面这样构建 sslCtx
SslContext sslCtx = SslContextBuilder.forClient().trustManager(InsecureTrustManagerFactory.INSTANCE).build();
并添加到管道p.addLast(sslCtx.newHandler(ch.alloc()));
最后我通过在代码中破解 TrustManager 解决了问题(在 SO 上找到了这个技术):
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0];
}
public void checkClientTrusted(X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(X509Certificate[] certs, String authType) {
}
}};
// Ignore differences between given hostname and certificate hostname
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier((hostname, session) -> true);
...
final SSLEngine sslEngine = sc.createSSLEngine(host, port);
sslEngine.setUseClientMode(true);
sslEngine.setNeedClientAuth(false);
p.addLast("ssl", new SslHandler(sslEngine));
现在所有在浏览器中正常的 https 站点(chrome 至少不显示警告)- 由 Netty 读取。
我有一些基于 netty (4.1b7) 的网络爬虫,我在其中大量请求不同的站点,包括 http 和 https 站点,并且我正在尝试配置 netty 客户端以处理具有不同身份验证设置的 https 站点。
当我有一个简单的 netty 配置时 w/o 自己的证书:
SslContext sslCtx = SslContextBuilder.forClient().build();
SSLEngine sslEngine = sslCtx.newEngine(ch.alloc(), host, port);
p.addLast("ssl", new SslHandler(sslEngine));
大约一半的 https 站点被请求成功,但其他站点失败:
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1728)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:304)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1506)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
at sun.security.ssl.Handshaker.run(Handshaker.java:919)
at sun.security.ssl.Handshaker.run(Handshaker.java:916)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1369)
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1164)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1067)
... 19 moreCaused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:387)
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:292)
at sun.security.validator.Validator.validate(Validator.java:260)
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324)
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:281)
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:136)
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1493)
... 27 more Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:146)
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:131)
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:382)
... 33 more
或:
Caused by: javax.net.ssl.SSLException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1138)
当我尝试 generate own local certificates 并将它们设置为:
System.setProperty("javax.net.ssl.trustStore", "/etc/ssl/my/cacerts.jks");
System.setProperty("javax.net.ssl.trustStorePassword", "changeit");
System.setProperty("javax.net.ssl.keyStore", "/etc/ssl/my/keystore.jks");
System.setProperty("javax.net.ssl.keyStorePassword", "changeit");
然后所有 https 站点都失败并出现如下错误:
Caused by: java.security.cert.CertificateException: found no certificates: /etc/ssl/my/cacerts.jks
at io.netty.handler.ssl.PemReader.readCertificates(PemReader.java:83) ~[netty-all-4.1.0.Beta7.jar:4.1.0.Beta7]
at io.netty.handler.ssl.SslContext.toX509Certificates(SslContext.java:967)
....
Caused by: java.security.KeyException: found no private key: /etc/ssl/my/keystore.jks
at io.netty.handler.ssl.PemReader.readPrivateKey(PemReader.java:99) ~[netty-all-4.1.0.Beta7.jar:4.1.0.Beta7]
at io.netty.handler.ssl.SslContext.toPrivateKey(SslContext.java:923)
我也尝试了建议 from that SO,但还没有成功。
出了什么问题或者任何人都可以提供某种逐步指南来配置 netty 4+ 客户端以处理具有所有可能的身份验证设置的 https 站点。
尝试像下面这样构建 sslCtx
SslContext sslCtx = SslContextBuilder.forClient().trustManager(InsecureTrustManagerFactory.INSTANCE).build();
并添加到管道p.addLast(sslCtx.newHandler(ch.alloc()));
最后我通过在代码中破解 TrustManager 解决了问题(在 SO 上找到了这个技术):
TrustManager[] trustAllCerts = new TrustManager[]{
new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() {
return new X509Certificate[0];
}
public void checkClientTrusted(X509Certificate[] certs, String authType) {
}
public void checkServerTrusted(X509Certificate[] certs, String authType) {
}
}};
// Ignore differences between given hostname and certificate hostname
SSLContext sc = SSLContext.getInstance("SSL");
sc.init(null, trustAllCerts, new SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sc.getSocketFactory());
HttpsURLConnection.setDefaultHostnameVerifier((hostname, session) -> true);
...
final SSLEngine sslEngine = sc.createSSLEngine(host, port);
sslEngine.setUseClientMode(true);
sslEngine.setNeedClientAuth(false);
p.addLast("ssl", new SslHandler(sslEngine));
现在所有在浏览器中正常的 https 站点(chrome 至少不显示警告)- 由 Netty 读取。