在为 spring 启动应用程序提供服务时，如何调试进入崩溃循环的无法解释的 Openshift 3 pod？
How can I debug this unexplained Openshift 3 pod going into crashloop when serving a spring boot app?
我正在尝试将 Spring 引导应用程序移动到 openshift 3 集群上。我正在使用 Maven fabric8 插件生成大部分 openshift 样板配置以及执行 S2I 构建。当我的 pod 启动时，我可以在日志输出中看到应用程序启动但在 spring 启动后立即默认为默认配置文件（我还没有设置配置文件）应用程序崩溃并且我看到的唯一输出在 Openshift 日志中被杀死......我找不到任何有价值的谷歌搜索，除了它似乎可能是 openJDK 试图获取比单个 pod 可用的内存更多的内存。我添加了一个 fabric8 片段来限制单个容器能够使用的内存，但是当我启动 pod 时我仍然遇到同样的错误。我 运行 OC describe pod 并看到退出代码 143。我 运行 没有想法，将不胜感激 关于如何调试此 further/how 以解决此类问题的任何想法？也不确定这是否相关，但即使我的 application.yml 设置为启用 SSL，fabric8 创建的路由始终是 HTTP URL 而不是 HTTPS URL。我想知道这是否可能是 pod 进入崩溃循环的原因，因为 readinessProbe 和 wellnessProbe 无法到达执行器端点？
下面是控制台输出、maven pom 文件、application.yml 和 fabric8 配置。
console output:
. ____ _ __ _ _
/\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.3.RELEASE)
2020-06-27 10:14:06.297 INFO 8 --- [ main] com.xib.vodacom.ans.AnsApplicationKt : Starting AnsApplicationKt v1.0.9 on vc-ans-1-bzlxc-debug with PID 8 (/deployments/vc-ans-1.0.9.jar started by ? in /deployments)
2020-06-27 10:14:06.396 DEBUG 8 --- [ main] com.xib.vodacom.ans.AnsApplicationKt : Running with Spring Boot v2.1.3.RELEASE, Spring v5.1.4.RELEASE
2020-06-27 10:14:06.397 INFO 8 --- [ main] com.xib.vodacom.ans.AnsApplicationKt : No active profile set, falling back to default profiles: default
Killed
sh-4.2$
oc describe pod output:
Secret (a volume populated by a Secret)
SecretName: certs
Optional: false
default-token-8pts4:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8pts4
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/compute=true
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m default-scheduler Successfully assigned cx-ans-prod/vc-ans-1-bzlxc to pocpap8zadsas.cc.corp
Normal Created 5m (x3 over 6m) kubelet, asdf.cc.corp Created container
Normal Started 5m (x3 over 6m) kubelet, asdfgf.cc.corp Started container
Normal Killing 5m (x2 over 6m) kubelet, asdfgf.cc.corp Killing container with id docker://spring-boot:Container failed liveness probe.. Container will be killed and recreated.
Warning Unhealthy 5m (x7 over 6m) kubelet, asdfgf.cc.corp Readiness probe failed: Get https://192.168.31.110:8080/actuator/health: dial tcp 192.168.31.110:8080: connect: connection refused
Warning Unhealthy 5m (x7 over 6m) kubelet, asdfgf.cc.corp Liveness probe failed: Get https://192.168.31.110:8080/actuator/health: dial tcp 192.168.31.110:8080: connect: connection refused
Normal Pulled 1m (x7 over 6m) kubelet, asdfgf.cc.corp Container image "docker-registry.default.svc:5000/cx-ans-prod/vc-ans@sha256:4f39366bddc1e4ce7ed3c7e320453fbe6e90c4100f860a82ec87b04e7fb7e5b1" already present on machine
C:\dev\xib-ans-bo>
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>com.xib</groupId>
<artifactId>vc-ans</artifactId>
<version>1.0.9</version>
<packaging>jar</packaging>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.3.RELEASE</version>
</parent>
<scm>
<connection>scm:git:git://pgitap1zatcrh.vodacom.corp:XIB/xib-ans-bo.git</connection>
<url>git://pgitap1zatcrh.vodacom.corp:XIB/xib-ans-bo.git</url>
</scm>
<properties>
<java.version>1.8</java.version>
<kotlin.version>1.3.61</kotlin.version>
<fabric8.mode>openshift</fabric8.mode>
<fabric8.namespace>cx-ans-prod</fabric8.namespace>
</properties>
<dependencies>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.3</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-data</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>java-jwt</artifactId>
<version>3.8.1</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-kotlin</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>25.1-jre</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
<version>1.3.21</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
<version>1.3.21</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.6</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<artifactId>ojdbc8</artifactId>
<groupId>com.oracle.ojdbc</groupId>
<version>19.3.0.0</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core</artifactId>
<version>1.0.0</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>*</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-core</artifactId>
<version>2.0.3.RELEASE</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-mockmvc</artifactId>
<version>2.0.3.RELEASE</version>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jre7</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jre8</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk7</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-runtime</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>2.1.2.RELEASE</version>
<scope>import</scope>
<type>pom</type>
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>java-jwt</artifactId>
<version>3.8.1</version>
<scope>compile</scope>
<type>pom</type>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<sourceDirectory>${project.basedir}/src/main/kotlin</sourceDirectory>
<testSourceDirectory>${project.basedir}/src/test/kotlin</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<configuration>
<args>
<arg>-Xjsr305=strict</arg>
</args>
<compilerPlugins>
<plugin>spring</plugin>
</compilerPlugins>
</configuration>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-allopen</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<executable>true</executable>
</configuration>
</plugin>
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>fabric8-maven-plugin</artifactId>
<version>4.4.1</version>
<configuration>
<generateRoute>true</generateRoute>
<enricher>
<config>
<fmp-service>
<name>ans-service</name>
<type>ClusterIP</type>
</fmp-service>
</config>
</enricher>
<enricher>
<config>
<fmp-maven-scm-enricher>>
<scm-tag>https</scm-tag>
</fmp-maven-scm-enricher>>
</config>
</enricher>
</configuration>
<executions>
<execution>
<goals>
<goal>resource</goal>
<goal>build</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
fabric8 deployment.yml fragment:
spec:
template:
spec:
containers:
- env:
- name: SPRING_PROFILES_ACTIVE
value: qa
volumeMounts:
- name: certs
mountPath: /opt/certs
readOnly: true
livenessProbe:
httpGet:
path: /actuator/health
port: 8080
scheme: HTTPS
initialDelaySeconds: 30
readinessProbe:
httpGet:
path: /actuator/health
port: 8080
scheme: HTTPS
initialDelaySeconds: 30
resources:
requests:
memory: "64Mi"
limits:
memory: "256Mi"
env:
- name: JVM_OPTS
value: "-Xms64M -Xmx256M"
volumes:
- name: certs
secret:
secretName: certs
items:
- key: ans.pfx
path: ans.pfx
- key: vb_pp_client_ans.jks
path: vb_pp_client_ans.jks
- key: vc_truststore.jks
path: vc_truststore.jks
spring boot application.yml:
app:
datasource:
driver-class-name: oracle.jdbc.driver.OracleDriver
url: jdbc:oracle:thin:@qgeno.cc.corp
username: asdfgf
password: asdfgf
connection-test-query: SELECT 1 from dual
pool-name: ANS_CP
minimum-idle: 3
maximum-pool-size: 20
alive:
file: /opt/apps/xib/vc-ans/running.txt
spring:
main:
allow-bean-definition-overriding: true
vb:
vb-url: "https://asdfgf.cc.corp"
hc-check-msisdn: "0821341111"
ssl-cert-password: "asdfgf"
ssl-cert-alias: "asdfgf"
keystore-path: "/opt/certs/vb_pp_client_ans.jks"
repeat-caller-url: https://asdfgf.cc.corp
server:
http:
port: 8080
ssl:
enabled: true
protocol: TLS
key-alias: lkjhj
key-store: /opt/certs/ans.pfx
key-store-password: asdfgf
trust-store: /opt/certs/vc_truststore.jks
trust-store-password: asdfgf
logging:
level:
com.xib: DEBUG
management:
server:
ssl:
key-store: /opt/certs/ans.pfx
key-store-password: asdfgf
endpoint:
health:
show-details: always
在下面添加application.yml
management.endpoint.health.group.readiness.include=*
management.endpoint.health.group.readiness.show-details=always
management.endpoint.health.group.liveness.include=ping
management.endpoint.health.group.liveness.show-details=never
在 deployment.yaml
中使用 /actuator/health/readiness 进行准备，使用 /actuator/health/liveness 进行活动
readinessProbe:
httpGet:
port: healthcheck
path: /actuator/health/readiness
initialDelaySeconds: 10
livenessProbe:
httpGet:
port: healthcheck
path: /actuator/health/liveness
initialDelaySeconds: 60
periodSeconds: 1
此外，如果存在 Spring 安全性，您将需要添加允许未经身份验证访问端点的自定义安全配置，如以下示例所示：
@Configuration(proxyBeanMethods = false)
public class ActuatorSecurity extends WebSecurityConfigurerAdapter {
@Override
protected void configure(HttpSecurity http) throws Exception {
http.requestMatcher(EndpointRequest.toAnyEndpoint()).authorizeRequests((requests) ->
requests.anyRequest().permitAll());
}
}
如果您在防火墙后部署应用程序，您可能希望无需身份验证即可访问所有执行器端点。您可以通过更改 management.endpoints.web.exposure.include 属性 来实现，如下
application.yaml
management.endpoints.web.exposure.include=*
错误代码 143 是 SIGTERM，Kubernetes 用来关闭 pods。这意味着 Kubernetes 出于某种原因希望您的 pod 被杀死。我认为可能有 3 个选项：
1：正如你所说，它可以由大于给定内存限制的内存分配触发。在那种情况下，Kubernetes 只会杀死你的 pod 并保持节点稳定，但如果它是 cpu 限制，那么 Kubernetes 可以限制资源分配并且不会杀死 pod。您可以通过 运行 此命令检查资源限制是否低于您的应用程序要求：oc get deployment deployment_name -o yaml
2：正如@Arghya Sadhu 所说，您可能以某种方式错误配置了执行器，请尝试检查是否正确设置了活性和就绪探针并提供了正确的输出。你也可以通过上面的命令获取这个配置。
3：除了您的应用需要太多准备时间并且 Kubernetes 认为您的应用行为不当外，一切都很好。如果您将资源限制设置得太窄并且您的 java 应用资源太少而无法及时准备，这可能会成为问题。
我正在尝试将 Spring 引导应用程序移动到 openshift 3 集群上。我正在使用 Maven fabric8 插件生成大部分 openshift 样板配置以及执行 S2I 构建。当我的 pod 启动时，我可以在日志输出中看到应用程序启动但在 spring 启动后立即默认为默认配置文件（我还没有设置配置文件）应用程序崩溃并且我看到的唯一输出在 Openshift 日志中被杀死......我找不到任何有价值的谷歌搜索，除了它似乎可能是 openJDK 试图获取比单个 pod 可用的内存更多的内存。我添加了一个 fabric8 片段来限制单个容器能够使用的内存，但是当我启动 pod 时我仍然遇到同样的错误。我 运行 OC describe pod 并看到退出代码 143。我 运行 没有想法，将不胜感激 关于如何调试此 further/how 以解决此类问题的任何想法？也不确定这是否相关，但即使我的 application.yml 设置为启用 SSL，fabric8 创建的路由始终是 HTTP URL 而不是 HTTPS URL。我想知道这是否可能是 pod 进入崩溃循环的原因，因为 readinessProbe 和 wellnessProbe 无法到达执行器端点？
下面是控制台输出、maven pom 文件、application.yml 和 fabric8 配置。
console output:
. ____ _ __ _ _
/\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.3.RELEASE)
2020-06-27 10:14:06.297 INFO 8 --- [ main] com.xib.vodacom.ans.AnsApplicationKt : Starting AnsApplicationKt v1.0.9 on vc-ans-1-bzlxc-debug with PID 8 (/deployments/vc-ans-1.0.9.jar started by ? in /deployments)
2020-06-27 10:14:06.396 DEBUG 8 --- [ main] com.xib.vodacom.ans.AnsApplicationKt : Running with Spring Boot v2.1.3.RELEASE, Spring v5.1.4.RELEASE
2020-06-27 10:14:06.397 INFO 8 --- [ main] com.xib.vodacom.ans.AnsApplicationKt : No active profile set, falling back to default profiles: default
Killed
sh-4.2$
oc describe pod output:
Secret (a volume populated by a Secret)
SecretName: certs
Optional: false
default-token-8pts4:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-8pts4
Optional: false
QoS Class: Burstable
Node-Selectors: node-role.kubernetes.io/compute=true
Tolerations: node.kubernetes.io/memory-pressure:NoSchedule
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m default-scheduler Successfully assigned cx-ans-prod/vc-ans-1-bzlxc to pocpap8zadsas.cc.corp
Normal Created 5m (x3 over 6m) kubelet, asdf.cc.corp Created container
Normal Started 5m (x3 over 6m) kubelet, asdfgf.cc.corp Started container
Normal Killing 5m (x2 over 6m) kubelet, asdfgf.cc.corp Killing container with id docker://spring-boot:Container failed liveness probe.. Container will be killed and recreated.
Warning Unhealthy 5m (x7 over 6m) kubelet, asdfgf.cc.corp Readiness probe failed: Get https://192.168.31.110:8080/actuator/health: dial tcp 192.168.31.110:8080: connect: connection refused
Warning Unhealthy 5m (x7 over 6m) kubelet, asdfgf.cc.corp Liveness probe failed: Get https://192.168.31.110:8080/actuator/health: dial tcp 192.168.31.110:8080: connect: connection refused
Normal Pulled 1m (x7 over 6m) kubelet, asdfgf.cc.corp Container image "docker-registry.default.svc:5000/cx-ans-prod/vc-ans@sha256:4f39366bddc1e4ce7ed3c7e320453fbe6e90c4100f860a82ec87b04e7fb7e5b1" already present on machine
C:\dev\xib-ans-bo>
pom.xml:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<modelVersion>4.0.0</modelVersion>
<groupId>com.xib</groupId>
<artifactId>vc-ans</artifactId>
<version>1.0.9</version>
<packaging>jar</packaging>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.1.3.RELEASE</version>
</parent>
<scm>
<connection>scm:git:git://pgitap1zatcrh.vodacom.corp:XIB/xib-ans-bo.git</connection>
<url>git://pgitap1zatcrh.vodacom.corp:XIB/xib-ans-bo.git</url>
</scm>
<properties>
<java.version>1.8</java.version>
<kotlin.version>1.3.61</kotlin.version>
<fabric8.mode>openshift</fabric8.mode>
<fabric8.namespace>cx-ans-prod</fabric8.namespace>
</properties>
<dependencies>
<dependency>
<groupId>commons-lang</groupId>
<artifactId>commons-lang</artifactId>
<version>2.3</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-data</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.security</groupId>
<artifactId>spring-security-test</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>java-jwt</artifactId>
<version>3.8.1</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-commons</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.module</groupId>
<artifactId>jackson-module-kotlin</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>25.1-jre</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.liquibase</groupId>
<artifactId>liquibase-core</artifactId>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
<version>1.3.21</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
<version>1.3.21</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.6</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<artifactId>ojdbc8</artifactId>
<groupId>com.oracle.ojdbc</groupId>
<version>19.3.0.0</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlinx</groupId>
<artifactId>kotlinx-coroutines-core</artifactId>
<version>1.0.0</version>
<scope>compile</scope>
<exclusions>
<exclusion>
<artifactId>tomcat-annotations-api</artifactId>
<groupId>org.apache.tomcat</groupId>
</exclusion>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>commons-logging</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>commons-logging</artifactId>
<groupId>*</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-core</artifactId>
<version>2.0.3.RELEASE</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.restdocs</groupId>
<artifactId>spring-restdocs-mockmvc</artifactId>
<version>2.0.3.RELEASE</version>
<scope>test</scope>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk8</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-reflect</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jre7</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jre8</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib-jdk7</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-stdlib</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-runtime</artifactId>
<version>1.3.21</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>2.1.2.RELEASE</version>
<scope>import</scope>
<type>pom</type>
</dependency>
<dependency>
<groupId>com.auth0</groupId>
<artifactId>java-jwt</artifactId>
<version>3.8.1</version>
<scope>compile</scope>
<type>pom</type>
</dependency>
</dependencies>
</dependencyManagement>
<build>
<sourceDirectory>${project.basedir}/src/main/kotlin</sourceDirectory>
<testSourceDirectory>${project.basedir}/src/test/kotlin</testSourceDirectory>
<plugins>
<plugin>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-plugin</artifactId>
<configuration>
<args>
<arg>-Xjsr305=strict</arg>
</args>
<compilerPlugins>
<plugin>spring</plugin>
</compilerPlugins>
</configuration>
<dependencies>
<dependency>
<groupId>org.jetbrains.kotlin</groupId>
<artifactId>kotlin-maven-allopen</artifactId>
<version>${kotlin.version}</version>
</dependency>
</dependencies>
</plugin>
<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
<configuration>
<executable>true</executable>
</configuration>
</plugin>
<plugin>
<groupId>io.fabric8</groupId>
<artifactId>fabric8-maven-plugin</artifactId>
<version>4.4.1</version>
<configuration>
<generateRoute>true</generateRoute>
<enricher>
<config>
<fmp-service>
<name>ans-service</name>
<type>ClusterIP</type>
</fmp-service>
</config>
</enricher>
<enricher>
<config>
<fmp-maven-scm-enricher>>
<scm-tag>https</scm-tag>
</fmp-maven-scm-enricher>>
</config>
</enricher>
</configuration>
<executions>
<execution>
<goals>
<goal>resource</goal>
<goal>build</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
fabric8 deployment.yml fragment:
spec:
template:
spec:
containers:
- env:
- name: SPRING_PROFILES_ACTIVE
value: qa
volumeMounts:
- name: certs
mountPath: /opt/certs
readOnly: true
livenessProbe:
httpGet:
path: /actuator/health
port: 8080
scheme: HTTPS
initialDelaySeconds: 30
readinessProbe:
httpGet:
path: /actuator/health
port: 8080
scheme: HTTPS
initialDelaySeconds: 30
resources:
requests:
memory: "64Mi"
limits:
memory: "256Mi"
env:
- name: JVM_OPTS
value: "-Xms64M -Xmx256M"
volumes:
- name: certs
secret:
secretName: certs
items:
- key: ans.pfx
path: ans.pfx
- key: vb_pp_client_ans.jks
path: vb_pp_client_ans.jks
- key: vc_truststore.jks
path: vc_truststore.jks
spring boot application.yml:
app:
datasource:
driver-class-name: oracle.jdbc.driver.OracleDriver
url: jdbc:oracle:thin:@qgeno.cc.corp
username: asdfgf
password: asdfgf
connection-test-query: SELECT 1 from dual
pool-name: ANS_CP
minimum-idle: 3
maximum-pool-size: 20
alive:
file: /opt/apps/xib/vc-ans/running.txt
spring:
main:
allow-bean-definition-overriding: true
vb:
vb-url: "https://asdfgf.cc.corp"
hc-check-msisdn: "0821341111"
ssl-cert-password: "asdfgf"
ssl-cert-alias: "asdfgf"
keystore-path: "/opt/certs/vb_pp_client_ans.jks"
repeat-caller-url: https://asdfgf.cc.corp
server:
http:
port: 8080
ssl:
enabled: true
protocol: TLS
key-alias: lkjhj
key-store: /opt/certs/ans.pfx
key-store-password: asdfgf
trust-store: /opt/certs/vc_truststore.jks
trust-store-password: asdfgf
logging:
level:
com.xib: DEBUG
management:
server:
ssl:
key-store: /opt/certs/ans.pfx
key-store-password: asdfgf
endpoint:
health:
show-details: always
在下面添加application.yml
management.endpoint.health.group.readiness.include=*
management.endpoint.health.group.readiness.show-details=always
management.endpoint.health.group.liveness.include=ping
management.endpoint.health.group.liveness.show-details=never
在 deployment.yaml
中使用/actuator/health/readiness 进行准备，使用 /actuator/health/liveness 进行活动
readinessProbe:
httpGet:
port: healthcheck
path: /actuator/health/readiness
initialDelaySeconds: 10
livenessProbe:
httpGet:
port: healthcheck
path: /actuator/health/liveness
initialDelaySeconds: 60
periodSeconds: 1
此外，如果存在 Spring 安全性，您将需要添加允许未经身份验证访问端点的自定义安全配置，如以下示例所示：
@Configuration(proxyBeanMethods = false)
public class ActuatorSecurity extends WebSecurityConfigurerAdapter {
@Override
protected void configure(HttpSecurity http) throws Exception {
http.requestMatcher(EndpointRequest.toAnyEndpoint()).authorizeRequests((requests) ->
requests.anyRequest().permitAll());
}
}
如果您在防火墙后部署应用程序，您可能希望无需身份验证即可访问所有执行器端点。您可以通过更改 management.endpoints.web.exposure.include 属性 来实现，如下
application.yaml
management.endpoints.web.exposure.include=*
错误代码 143 是 SIGTERM，Kubernetes 用来关闭 pods。这意味着 Kubernetes 出于某种原因希望您的 pod 被杀死。我认为可能有 3 个选项：
1：正如你所说，它可以由大于给定内存限制的内存分配触发。在那种情况下，Kubernetes 只会杀死你的 pod 并保持节点稳定，但如果它是 cpu 限制，那么 Kubernetes 可以限制资源分配并且不会杀死 pod。您可以通过 运行 此命令检查资源限制是否低于您的应用程序要求：oc get deployment deployment_name -o yaml
2：正如@Arghya Sadhu 所说，您可能以某种方式错误配置了执行器，请尝试检查是否正确设置了活性和就绪探针并提供了正确的输出。你也可以通过上面的命令获取这个配置。
3：除了您的应用需要太多准备时间并且 Kubernetes 认为您的应用行为不当外，一切都很好。如果您将资源限制设置得太窄并且您的 java 应用资源太少而无法及时准备，这可能会成为问题。