在 Jenkins Docker 代理中使用测试容器：容器无法启动，NoRouteToHostException
Using testcontainers in a Jenkins Docker Agent: containers fail to start, NoRouteToHostException
我正在使用带有 Docker 代理的 Jenkins 声明式管道来构建和测试我的软件，包括 运行 使用测试容器进行集成测试。我可以 运行 我的测试容器在我的开发环境中测试正常（不使用 Jenkins），但它们在 Jenkins 下失败了。
testcontainers Ryuk 资源收割守护进程不工作
16:29:20.255 [testcontainers-ryuk] WARN o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32769
java.net.NoRouteToHostException: No route to host (Host unreachable)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/java.net.Socket.connect(Socket.java:540)
at java.base/java.net.Socket.<init>(Socket.java:436)
at java.base/java.net.Socket.<init>(Socket.java:213)
at org.testcontainers.utility.ResourceReaper.lambda$start(ResourceReaper.java:112)
at java.base/java.lang.Thread.run(Thread.java:834)
通过将 TESTCONTAINERS_RYUK_DISABLED 环境变量设置为 true，我 disabling the daemon 解决了这个问题。但是，一些集成测试仍然反复失败。
使用 ElasticsearchContainer 的集成测试反复启动失败：等待 HTTP 端口响应超时。
17:04:57.595 [main] INFO d.e.c.7.1] - Starting container with ID: f5c653442103b9073c76f6ed91fc9117f7cb388d576606be8bd85bd9f3b2051d
17:04:58.465 [main] INFO d.e.c.7.1] - Container docker.elastic.co/elasticsearch/elasticsearch:6.7.1 is starting: f5c653442103b9073c76f6ed91fc9117f7cb388d576606be8bd85bd9f3b2051d
17:04:58.479 [main] INFO o.t.c.wait.strategy.HttpWaitStrategy - /loving_swartz: Waiting for 240 seconds for URL: http://172.17.0.1:32833/
17:08:58.480 [main] ERROR d.e.c.7.1] - Could not start container
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://172.17.0.1:32833/ should return HTTP 200)
at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:197)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:582)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:259)
at org.testcontainers.containers.GenericContainer.lambda$doStart[=12=](GenericContainer.java:212)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:210)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:199)
at
...
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
17:08:58.513 [main] ERROR d.e.c.7.1] - Log output from the failed container:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
[2019-04-11T17:05:02,527][INFO ][o.e.e.NodeEnvironment ] [1a_XhBT] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [1.2tb], net total_space [1.2tb], types [rootfs]
[2019-04-11T17:05:02,532][INFO ][o.e.e.NodeEnvironment ] [1a_XhBT] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-04-11T17:05:02,536][INFO ][o.e.n.Node ] [1a_XhBT] node name derived from node ID [1a_XhBTfQZWw1XLZMXrp4A]; set [node.name] to override
[2019-04-11T17:05:02,536][INFO ][o.e.n.Node ] [1a_XhBT] version[6.7.1], pid[1], build[default/docker/2f32220/2019-04-02T15:59:27.961366Z], OS[Linux/3.10.0-957.10.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[2019-04-11T17:05:02,536][INFO ][o.e.n.Node ] [1a_XhBT] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-14081126934203442674, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
...
[2019-04-11T17:05:16,338][INFO ][o.e.d.DiscoveryModule ] [1a_XhBT] using discovery type [single-node] and host providers [settings]
[2019-04-11T17:05:17,795][INFO ][o.e.n.Node ] [1a_XhBT] initialized
[2019-04-11T17:05:17,795][INFO ][o.e.n.Node ] [1a_XhBT] starting ...
[2019-04-11T17:05:18,086][INFO ][o.e.t.TransportService ] [1a_XhBT] publish_address {172.28.0.3:9300}, bound_addresses {0.0.0.0:9300}
[2019-04-11T17:05:18,128][WARN ][o.e.b.BootstrapChecks ] [1a_XhBT] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-04-11T17:05:18,299][INFO ][o.e.h.n.Netty4HttpServerTransport] [1a_XhBT] publish_address {172.28.0.3:9200}, bound_addresses {0.0.0.0:9200}
[2019-04-11T17:05:18,299][INFO ][o.e.n.Node ] [1a_XhBT] started
[2019-04-11T17:05:18,461][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [1a_XhBT] Failed to clear cache for realms [[]]
[2019-04-11T17:05:18,542][INFO ][o.e.g.GatewayService ] [1a_XhBT] recovered [0] indices into cluster_state
[2019-04-11T17:05:18,822][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2019-04-11T17:05:18,871][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.watches] for index patterns [.watches*]
[2019-04-11T17:05:18,906][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2019-04-11T17:05:18,955][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2019-04-11T17:05:19,017][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2019-04-11T17:05:19,054][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2019-04-11T17:05:19,100][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2019-04-11T17:05:19,148][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2019-04-11T17:05:19,480][INFO ][o.e.l.LicenseService ] [1a_XhBT] license [17853035-5cf6-49c8-96ca-4d14b26325f6] mode [basic] - valid
然而 Elasticsearch 日志文件看起来没问题，并且包括 Elasticsearch 在启动期间写入的最后一条日志消息（关于许可证）。
手动更改该容器以使用 HostPortWaitStrategy 而不是默认的 HttpWaitStrategy 没有帮助。
在尝试调查或解决此问题时，我更改了测试代码以通过为 testcontainers Network 对象调用 network.getId() 来显式启动 Docker 网络。然后失败并显示 NoRouteToHostException.
我该如何解决这个问题？
经过一些实验，我发现了问题的原因。关键操作是尝试在 Docker 容器内创建 Docker 桥接网络（使用 docker network create 或 testcontainers Network 对象） 自身 运行 在 Docker 桥接网络中。如果这样做，您将不会从 Docker 收到错误消息，Docker 守护程序日志文件也不会包含任何有用的消息。但是尝试使用网络将导致 "no route to host".
我通过让最外层的 Docker 容器（Jenkins 代理）访问主机网络解决了这个问题，方法是让 Jenkins 为其 docker run 命令提供 --network="host" 选项：
pipeline {
agent {
dockerfile {
filename 'Dockerfile.jenkinsAgent'
additionalBuildArgs ...
args '-v /var/run/docker.sock:/var/run/docker.sock ... --network="host" -u jenkins:docker'
}
}
stages {
...
没关系，因为 Jenkins 代理不需要桥接网络提供的隔离级别。
在我的例子中，向 Docker 代理选项添加两个参数就足够了：
- Docker socket as volume
- 添加 ID 为 docker 组的
--group-add 参数
pipeline {
agent any
stages {
stage('Gradle build') {
agent {
docker {
reuseNode true
image 'openjdk:11.0-jdk-slim'
args '-v /var/run/docker.sock:/var/run/docker.sock --group-add 992'
}
}
steps {
sh 'env | sort'
sh './gradlew build --no-daemon --stacktrace'
}
}
} // stages
} // pipeline
我正在使用带有 Docker 代理的 Jenkins 声明式管道来构建和测试我的软件，包括 运行 使用测试容器进行集成测试。我可以 运行 我的测试容器在我的开发环境中测试正常（不使用 Jenkins），但它们在 Jenkins 下失败了。
testcontainers Ryuk 资源收割守护进程不工作
16:29:20.255 [testcontainers-ryuk] WARN o.t.utility.ResourceReaper - Can not connect to Ryuk at 172.17.0.1:32769
java.net.NoRouteToHostException: No route to host (Host unreachable)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/java.net.Socket.connect(Socket.java:540)
at java.base/java.net.Socket.<init>(Socket.java:436)
at java.base/java.net.Socket.<init>(Socket.java:213)
at org.testcontainers.utility.ResourceReaper.lambda$start(ResourceReaper.java:112)
at java.base/java.lang.Thread.run(Thread.java:834)
通过将 TESTCONTAINERS_RYUK_DISABLED 环境变量设置为 true，我 disabling the daemon 解决了这个问题。但是，一些集成测试仍然反复失败。
使用 ElasticsearchContainer 的集成测试反复启动失败：等待 HTTP 端口响应超时。
17:04:57.595 [main] INFO d.e.c.7.1] - Starting container with ID: f5c653442103b9073c76f6ed91fc9117f7cb388d576606be8bd85bd9f3b2051d
17:04:58.465 [main] INFO d.e.c.7.1] - Container docker.elastic.co/elasticsearch/elasticsearch:6.7.1 is starting: f5c653442103b9073c76f6ed91fc9117f7cb388d576606be8bd85bd9f3b2051d
17:04:58.479 [main] INFO o.t.c.wait.strategy.HttpWaitStrategy - /loving_swartz: Waiting for 240 seconds for URL: http://172.17.0.1:32833/
17:08:58.480 [main] ERROR d.e.c.7.1] - Could not start container
org.testcontainers.containers.ContainerLaunchException: Timed out waiting for URL to be accessible (http://172.17.0.1:32833/ should return HTTP 200)
at org.testcontainers.containers.wait.strategy.HttpWaitStrategy.waitUntilReady(HttpWaitStrategy.java:197)
at org.testcontainers.containers.wait.strategy.AbstractWaitStrategy.waitUntilReady(AbstractWaitStrategy.java:35)
at org.testcontainers.containers.GenericContainer.waitUntilContainerStarted(GenericContainer.java:582)
at org.testcontainers.containers.GenericContainer.tryStart(GenericContainer.java:259)
at org.testcontainers.containers.GenericContainer.lambda$doStart[=12=](GenericContainer.java:212)
at org.rnorth.ducttape.unreliables.Unreliables.retryUntilSuccess(Unreliables.java:76)
at org.testcontainers.containers.GenericContainer.doStart(GenericContainer.java:210)
at org.testcontainers.containers.GenericContainer.start(GenericContainer.java:199)
at
...
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
17:08:58.513 [main] ERROR d.e.c.7.1] - Log output from the failed container:
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
[2019-04-11T17:05:02,527][INFO ][o.e.e.NodeEnvironment ] [1a_XhBT] using [1] data paths, mounts [[/ (rootfs)]], net usable_space [1.2tb], net total_space [1.2tb], types [rootfs]
[2019-04-11T17:05:02,532][INFO ][o.e.e.NodeEnvironment ] [1a_XhBT] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-04-11T17:05:02,536][INFO ][o.e.n.Node ] [1a_XhBT] node name derived from node ID [1a_XhBTfQZWw1XLZMXrp4A]; set [node.name] to override
[2019-04-11T17:05:02,536][INFO ][o.e.n.Node ] [1a_XhBT] version[6.7.1], pid[1], build[default/docker/2f32220/2019-04-02T15:59:27.961366Z], OS[Linux/3.10.0-957.10.1.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/12/12+33]
[2019-04-11T17:05:02,536][INFO ][o.e.n.Node ] [1a_XhBT] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=/tmp/elasticsearch-14081126934203442674, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Djava.locale.providers=COMPAT, -XX:UseAVX=2, -Des.cgroups.hierarchy.override=/, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=default, -Des.distribution.type=docker]
...
[2019-04-11T17:05:16,338][INFO ][o.e.d.DiscoveryModule ] [1a_XhBT] using discovery type [single-node] and host providers [settings]
[2019-04-11T17:05:17,795][INFO ][o.e.n.Node ] [1a_XhBT] initialized
[2019-04-11T17:05:17,795][INFO ][o.e.n.Node ] [1a_XhBT] starting ...
[2019-04-11T17:05:18,086][INFO ][o.e.t.TransportService ] [1a_XhBT] publish_address {172.28.0.3:9300}, bound_addresses {0.0.0.0:9300}
[2019-04-11T17:05:18,128][WARN ][o.e.b.BootstrapChecks ] [1a_XhBT] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2019-04-11T17:05:18,299][INFO ][o.e.h.n.Netty4HttpServerTransport] [1a_XhBT] publish_address {172.28.0.3:9200}, bound_addresses {0.0.0.0:9200}
[2019-04-11T17:05:18,299][INFO ][o.e.n.Node ] [1a_XhBT] started
[2019-04-11T17:05:18,461][WARN ][o.e.x.s.a.s.m.NativeRoleMappingStore] [1a_XhBT] Failed to clear cache for realms [[]]
[2019-04-11T17:05:18,542][INFO ][o.e.g.GatewayService ] [1a_XhBT] recovered [0] indices into cluster_state
[2019-04-11T17:05:18,822][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.watch-history-9] for index patterns [.watcher-history-9*]
[2019-04-11T17:05:18,871][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.watches] for index patterns [.watches*]
[2019-04-11T17:05:18,906][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.triggered_watches] for index patterns [.triggered_watches*]
[2019-04-11T17:05:18,955][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-logstash] for index patterns [.monitoring-logstash-6-*]
[2019-04-11T17:05:19,017][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-es] for index patterns [.monitoring-es-6-*]
[2019-04-11T17:05:19,054][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-alerts] for index patterns [.monitoring-alerts-6]
[2019-04-11T17:05:19,100][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-beats] for index patterns [.monitoring-beats-6-*]
[2019-04-11T17:05:19,148][INFO ][o.e.c.m.MetaDataIndexTemplateService] [1a_XhBT] adding template [.monitoring-kibana] for index patterns [.monitoring-kibana-6-*]
[2019-04-11T17:05:19,480][INFO ][o.e.l.LicenseService ] [1a_XhBT] license [17853035-5cf6-49c8-96ca-4d14b26325f6] mode [basic] - valid
然而 Elasticsearch 日志文件看起来没问题，并且包括 Elasticsearch 在启动期间写入的最后一条日志消息（关于许可证）。
手动更改该容器以使用 HostPortWaitStrategy 而不是默认的 HttpWaitStrategy 没有帮助。
在尝试调查或解决此问题时，我更改了测试代码以通过为 testcontainers Network 对象调用 network.getId() 来显式启动 Docker 网络。然后失败并显示 NoRouteToHostException.
我该如何解决这个问题？
经过一些实验，我发现了问题的原因。关键操作是尝试在 Docker 容器内创建 Docker 桥接网络（使用 docker network create 或 testcontainers Network 对象） 自身 运行 在 Docker 桥接网络中。如果这样做，您将不会从 Docker 收到错误消息，Docker 守护程序日志文件也不会包含任何有用的消息。但是尝试使用网络将导致 "no route to host".
我通过让最外层的 Docker 容器（Jenkins 代理）访问主机网络解决了这个问题，方法是让 Jenkins 为其 docker run 命令提供 --network="host" 选项：
pipeline {
agent {
dockerfile {
filename 'Dockerfile.jenkinsAgent'
additionalBuildArgs ...
args '-v /var/run/docker.sock:/var/run/docker.sock ... --network="host" -u jenkins:docker'
}
}
stages {
...
没关系，因为 Jenkins 代理不需要桥接网络提供的隔离级别。
在我的例子中，向 Docker 代理选项添加两个参数就足够了：
- Docker socket as volume
- 添加 ID 为 docker 组的
--group-add参数
pipeline {
agent any
stages {
stage('Gradle build') {
agent {
docker {
reuseNode true
image 'openjdk:11.0-jdk-slim'
args '-v /var/run/docker.sock:/var/run/docker.sock --group-add 992'
}
}
steps {
sh 'env | sort'
sh './gradlew build --no-daemon --stacktrace'
}
}
} // stages
} // pipeline