hostPathVolume 未安装在 Kubernetes pods（动态 Jenkins 代理）上
hostPathVolume not mounting on Kubernetes pods(Dynamic Jenkins Agent)
Environment:
Jenkins - Version 2.235.5 (LTS)
Kubernetes Jenkins Plugin - 1.27.7
K8s-Master & Worker Server Details:-
OS - CentOS Linux release 7.9.2009 (Core) & Kernal - 3.10.0-1160.6.1.el7.x86_64
Docker Version - 19.03.12
Client Version: v1.19.0
Server Version: v1.19.0
我正在尝试通过 Jenkins Kubernetes-plugin podTemplate 安装一个 hostPathVolume 并且当 pod 通过 Jenkins master 启动时我的 K8s worker 的路径没有安装在 K8s pods(Jenkins 代理).
我有以下脚本管道。
podTemplate(label: 'mypod', containers: [
containerTemplate(name: 'git', image: 'ubuntu/16.04', ttyEnabled: true, command: 'cat'),
containerTemplate(name: 'maven', image: 'ubuntu/16.04', command: 'cat', ttyEnabled: true)
],
volumes: [
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
]
) {
node('mypod') {
stage('Check running containers') {
container('docker') {
// example to show you can run docker commands when you mount the socket
sh 'hostname'
sh 'hostname -i'
sh 'docker ps'
}
}
..... further stages....
K8s集群Jenkins代理pod日志
# kubectl logs -f jenkins_job-10-b04wk-dn4gf-llpbp -c jnlp
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: jenkins_job-10-b04wk-dn4gf-llpbp
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Dec 05, 2020 5:41:16 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins-server/jenkins/]
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-server
Agent port: 40003
Identity: 23:66:a9:70:0b:f2:03:e0:47:44:0d:ee:7e:ce:8c:31
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-server:40003
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 23:66:a9:70:0b:f2:03:e0:47:44:0d:ee:7e:ce:8c:31
Dec 05, 2020 5:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Dec 05, 2020 5:41:21 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call
INFO: Disabled slave engine reconnects.
Dec 05, 2020 5:41:21 AM hudson.util.ProcessTree get
WARNING: Error while determining if vetoers exist
hudson.remoting.RequestAbortedException: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. It is not open for write
at hudson.remoting.Request.abort(Request.java:340)
at hudson.remoting.Channel.terminate(Channel.java:1081)
at hudson.remoting.Channel.close(Channel.java:1482)
at hudson.remoting.Channel.close(Channel.java:1446)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1313)
at hudson.remoting.Channel.handle(Channel.java:606)
at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:203)
at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:189)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:187)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:206)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access00(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine.lambda$newThread[=13=](Engine.java:117)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkins-server/10.20.30.35:40003
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
at hudson.remoting.Request.call(Request.java:202)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
at com.sun.proxy.$Proxy5.fetch3(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:211)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at jenkins.agents.AgentComputerUtil.getChannelToMaster(AgentComputerUtil.java:48)
at hudson.util.ProcessTree.get(ProcessTree.java:432)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 4 more
Caused by: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. It is not open for write
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:305)
at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:304)
at hudson.remoting.Channel.send(Channel.java:764)
at hudson.remoting.Channel.close(Channel.java:1479)
... 18 more
Dec 05, 2020 5:41:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Dec 05, 2020 5:41:21 AM hudson.util.ProcessTree get
WARNING: Failed to load winp. Reverting to the default
java.lang.NoClassDefFoundError: hudson/util/ProcessTree$Linux$LinuxProcess
at hudson.util.ProcessTree.get(ProcessTree.java:451)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine.lambda$newThread[=13=](Engine.java:117)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$Linux$LinuxProcess
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:173)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 12 more
Dec 05, 2020 5:41:21 AM hudson.remoting.Request run
INFO: Failed to send back a reply to the request hudson.remoting.Request@1affe1cf: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@157a1c18:JNLP4-connect connection to jenkins-server/10.20.30.35:40003": channel is already closed
openjdk version "1.8.0_272" 安装在 (Jenkins master, K8s Master and K8s-Worker) 个系统中，也重新启动了所有三个系统，但仍然没有成功。
我希望卷安装在我的 pod 中，我已经尝试使用 K8s v1.19.4 和 Kubernetes Jenkins Plugin - 1.27.0。不确定是什么导致了 CentOS-7.9 上的问题。而在 K8s master 主机的 OS.
的 CentOS - 8.2.2004 (Core) & 4.18.0-193.14.2.el8_2.x86_64 内核上同样完美地工作
我已经将我的 docker 图片从 Ubuntu 16.04.6 LTS - 3.10.0-1160.6.1.el7.x86_64 升级到 Ubuntu 16.04.7 to 3.10.0-1160.6.1.el7.x86_64 仍然没有用。但似乎 Kubernetes on CentOS-7.9 & with Kernal-3.10.0-1160.6.1.el7.x86_64 存在内核特定问题。所以我已经将我的本地主机的 OS 升级到 CentOS-8.2 with - 4.18.0-193.28.1.el8_2.x86_64 内核，现在我的管道构建工作正常。
Environment:
Jenkins - Version 2.235.5 (LTS)
Kubernetes Jenkins Plugin - 1.27.7
K8s-Master & Worker Server Details:-
OS - CentOS Linux release 7.9.2009 (Core) & Kernal - 3.10.0-1160.6.1.el7.x86_64
Docker Version - 19.03.12
Client Version: v1.19.0
Server Version: v1.19.0
我正在尝试通过 Jenkins Kubernetes-plugin podTemplate 安装一个 hostPathVolume 并且当 pod 通过 Jenkins master 启动时我的 K8s worker 的路径没有安装在 K8s pods(Jenkins 代理).
我有以下脚本管道。
podTemplate(label: 'mypod', containers: [
containerTemplate(name: 'git', image: 'ubuntu/16.04', ttyEnabled: true, command: 'cat'),
containerTemplate(name: 'maven', image: 'ubuntu/16.04', command: 'cat', ttyEnabled: true)
],
volumes: [
hostPathVolume(mountPath: '/var/run/docker.sock', hostPath: '/var/run/docker.sock'),
]
) {
node('mypod') {
stage('Check running containers') {
container('docker') {
// example to show you can run docker commands when you mount the socket
sh 'hostname'
sh 'hostname -i'
sh 'docker ps'
}
}
..... further stages....
K8s集群Jenkins代理pod日志
# kubectl logs -f jenkins_job-10-b04wk-dn4gf-llpbp -c jnlp
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main createEngine
INFO: Setting up agent: jenkins_job-10-b04wk-dn4gf-llpbp
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener <init>
INFO: Jenkins agent is running in headless mode.
Dec 05, 2020 5:41:16 AM hudson.remoting.Engine startEngine
INFO: Using Remoting version: 4.3
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.WorkDirManager initializeWorkDir
INFO: Using /home/jenkins/agent/remoting as a remoting work directory
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.WorkDirManager setupLogging
INFO: Both error and output logs will be printed to /home/jenkins/agent/remoting
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://jenkins-server/jenkins/]
Dec 05, 2020 5:41:16 AM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
Agent address: jenkins-server
Agent port: 40003
Identity: 23:66:a9:70:0b:f2:03:e0:47:44:0d:ee:7e:ce:8c:31
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to jenkins-server:40003
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Trying protocol: JNLP4-connect
Dec 05, 2020 5:41:16 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Remote identity confirmed: 23:66:a9:70:0b:f2:03:e0:47:44:0d:ee:7e:ce:8c:31
Dec 05, 2020 5:41:17 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connected
Dec 05, 2020 5:41:21 AM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave$SlaveDisconnector call
INFO: Disabled slave engine reconnects.
Dec 05, 2020 5:41:21 AM hudson.util.ProcessTree get
WARNING: Error while determining if vetoers exist
hudson.remoting.RequestAbortedException: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. It is not open for write
at hudson.remoting.Request.abort(Request.java:340)
at hudson.remoting.Channel.terminate(Channel.java:1081)
at hudson.remoting.Channel.close(Channel.java:1482)
at hudson.remoting.Channel.close(Channel.java:1446)
at hudson.remoting.Channel$CloseCommand.execute(Channel.java:1313)
at hudson.remoting.Channel.handle(Channel.java:606)
at hudson.remoting.AbstractByteBufferCommandTransport.processCommand(AbstractByteBufferCommandTransport.java:203)
at hudson.remoting.AbstractByteBufferCommandTransport.receive(AbstractByteBufferCommandTransport.java:189)
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer.onRead(ChannelApplicationLayer.java:187)
at org.jenkinsci.remoting.protocol.ApplicationLayer.onRecv(ApplicationLayer.java:206)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.processRead(SSLEngineFilterLayer.java:369)
at org.jenkinsci.remoting.protocol.impl.SSLEngineFilterLayer.onRecv(SSLEngineFilterLayer.java:117)
at org.jenkinsci.remoting.protocol.ProtocolStack$Ptr.onRecv(ProtocolStack.java:668)
at org.jenkinsci.remoting.protocol.NetworkLayer.onRead(NetworkLayer.java:136)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer.access00(BIONetworkLayer.java:48)
at org.jenkinsci.remoting.protocol.impl.BIONetworkLayer$Reader.run(BIONetworkLayer.java:283)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine.lambda$newThread[=13=](Engine.java:117)
at java.lang.Thread.run(Thread.java:748)
Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection to jenkins-server/10.20.30.35:40003
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
at hudson.remoting.Request.call(Request.java:202)
at hudson.remoting.RemoteInvocationHandler.invoke(RemoteInvocationHandler.java:288)
at com.sun.proxy.$Proxy5.fetch3(Unknown Source)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:211)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at jenkins.agents.AgentComputerUtil.getChannelToMaster(AgentComputerUtil.java:48)
at hudson.util.ProcessTree.get(ProcessTree.java:432)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 4 more
Caused by: hudson.remoting.ChannelClosedException: Channel "unknown": Protocol stack cannot write data anymore. It is not open for write
at org.jenkinsci.remoting.protocol.impl.ChannelApplicationLayer$ByteBufferCommandTransport.write(ChannelApplicationLayer.java:305)
at hudson.remoting.AbstractByteBufferCommandTransport.write(AbstractByteBufferCommandTransport.java:304)
at hudson.remoting.Channel.send(Channel.java:764)
at hudson.remoting.Channel.close(Channel.java:1479)
... 18 more
Dec 05, 2020 5:41:21 AM hudson.remoting.jnlp.Main$CuiListener status
INFO: Terminated
Dec 05, 2020 5:41:21 AM hudson.util.ProcessTree get
WARNING: Failed to load winp. Reverting to the default
java.lang.NoClassDefFoundError: hudson/util/ProcessTree$Linux$LinuxProcess
at hudson.util.ProcessTree.get(ProcessTree.java:451)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1100)
at hudson.Launcher$RemoteLauncher$KillTask.call(Launcher.java:1091)
at hudson.remoting.UserRequest.perform(UserRequest.java:211)
at hudson.remoting.UserRequest.perform(UserRequest.java:54)
at hudson.remoting.Request.run(Request.java:369)
at hudson.remoting.InterceptingExecutorService.call(InterceptingExecutorService.java:72)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at hudson.remoting.Engine.lambda$newThread[=13=](Engine.java:117)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: hudson.util.ProcessTree$Linux$LinuxProcess
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at hudson.remoting.RemoteClassLoader.findClass(RemoteClassLoader.java:173)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
... 12 more
Dec 05, 2020 5:41:21 AM hudson.remoting.Request run
INFO: Failed to send back a reply to the request hudson.remoting.Request@1affe1cf: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@157a1c18:JNLP4-connect connection to jenkins-server/10.20.30.35:40003": channel is already closed
openjdk version "1.8.0_272" 安装在 (Jenkins master, K8s Master and K8s-Worker) 个系统中，也重新启动了所有三个系统，但仍然没有成功。
我希望卷安装在我的 pod 中，我已经尝试使用 K8s v1.19.4 和 Kubernetes Jenkins Plugin - 1.27.0。不确定是什么导致了 CentOS-7.9 上的问题。而在 K8s master 主机的 OS.
CentOS - 8.2.2004 (Core) & 4.18.0-193.14.2.el8_2.x86_64 内核上同样完美地工作
我已经将我的 docker 图片从 Ubuntu 16.04.6 LTS - 3.10.0-1160.6.1.el7.x86_64 升级到 Ubuntu 16.04.7 to 3.10.0-1160.6.1.el7.x86_64 仍然没有用。但似乎 Kubernetes on CentOS-7.9 & with Kernal-3.10.0-1160.6.1.el7.x86_64 存在内核特定问题。所以我已经将我的本地主机的 OS 升级到 CentOS-8.2 with - 4.18.0-193.28.1.el8_2.x86_64 内核，现在我的管道构建工作正常。