com.spotify.docker.client.DockerRequestException: Request error: DELETE unix://localhost:80/v1.12/containers/...: 409

com.spotify.docker.client.DockerRequestException: Request error: DELETE unix://localhost:80/v1.12/containers/...: 409

我正在开发一个 Java 应用程序,该应用程序使用 Google Kubernetes 在 Apache Tomcat Docker 容器中部署 Web 工件。我正在使用 https://github.com/spotify/docker-client in order to carry out Docker Image and Container handling activities and https://github.com/fabric8io/fabric8/tree/master/components/kubernetes-api 来实现与 Kubernetes 相关的功能。

在此应用程序中,我添加了一项功能,使用户能够删除用户部署的 Web 工件。

删除 I 时,

  1. 删除我用来生成所需数量的 pod 副本的 Kubernetes 复制控制器

  2. 单独删除replica pods(因为pods在Java中相应方法删除replication controller时不会自动删除API)

  3. 删除对应创建的Service

  4. 删除掉Docker容器对应的pods删除掉

  5. 最后,删除用于部署的 Docker 图像

以下代码显示了已实现的删除功能:

public boolean remove(String tenant, String appName) throws WebArtifactHandlerException {
        String componentName = generateKubernetesComponentName(tenant, appName);
        final int singleImageIndex = 0;
        try {
            if (replicationControllerHandler.getReplicationController(componentName) != null) {
                String dockerImage = replicationControllerHandler.getReplicationController(componentName).getSpec()
                        .getTemplate().getSpec().getContainers().get(singleImageIndex).getImage();
                List<String> containerIds = containerHandler.getRunningContainerIdsByImage(dockerImage);
                replicationControllerHandler.deleteReplicationController(componentName);
                podHandler.deleteReplicaPods(tenant, appName);
                serviceHandler.deleteService(componentName);
                Thread.sleep(OPERATION_DELAY_IN_MILLISECONDS);
                containerHandler.deleteContainers(containerIds);
                imageBuilder.removeImage(tenant, appName, getDockerImageVersion(dockerImage));
                return true;
            } else {
                return false;
            }
        } catch (Exception exception) {
            String message = String.format("Failed to remove web artifact[artifact]: %s",
                    generateKubernetesComponentName(tenant, appName));
            LOG.error(message, exception);
            throw new WebArtifactHandlerException(message, exception);
        }
    } 

Docker容器删除功能的实现如下:

public void deleteContainers(List<String> containerIds) throws WebArtifactHandlerException {
        try {
            for (String containerId : containerIds) {
                dockerClient.removeContainer(containerId);
                Thread.sleep(OPERATION_DELAY_IN_MILLISECONDS);
            }
        } catch (Exception exception) {
            String message = "Could not delete the Docker Containers.";
            LOG.error(message, exception);
            throw new WebArtifactHandlerException(message, exception);
        }
    }

在上述情况下,虽然所需函数的执行没有任何问题,但在某些情况下,我往往会遇到以下异常。

Sep 11, 2015 3:57:28 PM org.apache.poc.webartifact.WebArtifactHandler remove
SEVERE: Failed to remove web artifact[artifact]: app-wso2-com
org.apache.poc.miscellaneous.exceptions.WebArtifactHandlerException: Could not delete the Docker Containers.
    at org.apache.poc.docker.JavaWebArtifactContainerHandler.deleteContainers(JavaWebArtifactContainerHandler.java:80)
    at org.apache.poc.webartifact.WebArtifactHandler.remove(WebArtifactHandler.java:206)
    at org.apache.poc.Executor.process(Executor.java:222)
    at org.apache.poc.Executor.main(Executor.java:46)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: com.spotify.docker.client.DockerRequestException: Request error: DELETE unix://localhost:80/v1.12/containers/af05916d2bddf73dcf8bf41c6ea7f5f3b859c90b97447a8248ffa7b5b3968691: 409
    at com.spotify.docker.client.DefaultDockerClient.propagate(DefaultDockerClient.java:1061)
    at com.spotify.docker.client.DefaultDockerClient.request(DefaultDockerClient.java:1021)
    at com.spotify.docker.client.DefaultDockerClient.removeContainer(DefaultDockerClient.java:544)
    at com.spotify.docker.client.DefaultDockerClient.removeContainer(DefaultDockerClient.java:535)
    at org.wso2.carbon6.poc.docker.JavaWebArtifactContainerHandler.deleteContainers(JavaWebArtifactContainerHandler.java:74)
    ... 8 more
Caused by: com.spotify.docker.client.shaded.javax.ws.rs.ClientErrorException: HTTP 409 Conflict
    at org.glassfish.jersey.client.JerseyInvocation.createExceptionForFamily(JerseyInvocation.java:991)
    at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:975)
    at org.glassfish.jersey.client.JerseyInvocation.translate(JerseyInvocation.java:795)
    at org.glassfish.jersey.client.JerseyInvocation.access0(JerseyInvocation.java:91)
    at org.glassfish.jersey.client.JerseyInvocation.completed(JerseyInvocation.java:756)
    at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:189)
    at org.glassfish.jersey.client.ClientRuntime.access0(ClientRuntime.java:74)
    at org.glassfish.jersey.client.ClientRuntime.run(ClientRuntime.java:171)
    at org.glassfish.jersey.internal.Errors.call(Errors.java:271)
    at org.glassfish.jersey.internal.Errors.call(Errors.java:267)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:315)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:297)
    at org.glassfish.jersey.internal.Errors.process(Errors.java:267)
    at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:320)
    at org.glassfish.jersey.client.ClientRuntime.run(ClientRuntime.java:201)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

我搜索了大量资源以寻求任何帮助,但我仍然无法在所有情况下避免它,我执行了此功能。

一开始我比现在更容易遇到这个问题,但是允许执行线程在删除每个 Docker 容器结束时和删除任何 Docker 容器之前休眠,逐渐减少了我遇到这个问题的次数。

让线程休眠是这个问题的最终解决方案还是有任何其他原因导致这个问题出现以及可以帮助我避免此异常的解决方案?任何帮助是极大的赞赏。

很遗憾,我不熟悉 Java 客户端库。

我的建议是尝试使用常规命令行客户端 (kubectl)。如果可行,那么您就知道问题出在 Java 客户端库或您对它的使用。如果使用命令行客户端不起作用,那么会有更多人可以帮助您(因为熟悉命令行客户端的人比熟悉 Java 客户端库的人多得多)。

也就是说 % kubectl delete pods ... # --cascade=true 默认 % kubectl 删除服务 ...

我很好奇你为什么需要步骤 (4) 和 (5)。步骤 (4) 应该在您删除 pod 时自动执行,步骤 (5) 应该在后台自动执行。

如果 "kubectl delete" 的两行有效,则问题出在 Java 客户端库或您对它的使用。作为起点,我建议从 Java 代码中删除对 deleteContainers() 和 removeImage() 的调用,看看是否有帮助。我认为这些步骤是不必要的。