对于 Hazelcast sidecar 缓存模式,客户端是否会发现每个节点两次?

Is it to be expected that the client will discover each node twice for the Hazelcast sidecar caching pattern?

我对使用 Hazelcast 还很陌生,因为它具有与其他缓存实例自动同步的有趣功能。我的查询在描述的底部。

这是我最初的目标:

  1. 按照 Hazelcast sidecar 缓存 模式设计环境。
  2. 应用程序容器端将没有缓存。基本上,我不想使用“近缓存”只是为了避免我的 JVM 很重并减少 GC 时间。
  3. 每个 Node 中的应用程序容器 将通过本地主机 IP 与其自己的 sidecar 缓存容器通信。
  4. Hazelcast 管理中心 将是一个单独的节点,它与包含 Hazelcast sidecar 缓存 容器的所有节点进行通信。

目标设计如下:

我为 Hazelcast 容器准备了 Hazelcast 配置 [hazelcast.yaml],

    hazelcast:
    cluster-name: dev
    network:
    port:
      auto-increment: false
      port-count: 3
      port: 5701

我还为我的应用程序容器准备了另一个hazelcast.yaml,

    hazelcast:
      map:
        default:
          backup-count: 0
          async-backup-count: 1
          read-backup-data: true
      network:
        reuse-address: true
        port:
          auto-increment: true
          port: 5701
        join:
          multicast:
            enabled: true
          kubernetes:
            enabled: false
          tcp-ip:
            enabled: false
            interaface: 127.0.0.1
            member-list:
              - 127.0.0.1:5701

这里是客户端部分,我用的是SpringBoot

    @Component
    public class CacheClient {
        
        private static final String ITEMS = "items";
        
        private HazelcastInstance client;

        CacheClient() throws IOException {
            ClientConfig config = new YamlClientConfigBuilder("hazelcast.yaml").build();
            config.setInstanceName(UUID.randomUUID().toString());
            client = HazelcastClient.getOrCreateHazelcastClient(config);
        }
    
        public Item put(String number, Item item){
            IMap<String, Item> map = client.getMap(ITEMS);
            return map.putIfAbsent(number, item);
        }
    
        public Item get(String key){
            IMap<String, Item> map = client.getMap(ITEMS);
            return map.get(key);
        }
    }

这是我用来构建我的应用容器镜像的dockerfile,

    FROM adoptopenjdk/openjdk11:jdk-11.0.5_10-alpine-slim
    # Expose port 8081 to Docker host
    EXPOSE 8081
    WORKDIR /opt
    COPY /build/libs/hazelcast-client-0.0.1-SNAPSHOT.jar /opt/app.jar
    COPY /src/main/resources/hazelcast.yaml /opt/hazelcast.yaml
    COPY /src/main/resources/application.properties /opt/application.properties
    ENTRYPOINT ["java","-Dhazelcast.socket.server.bind.any=false","-Dhazelcast.initial.min.cluster.size=1","-Dhazelcast.socket.bind.any=false","-Dhazelcast.socket.server.bind.any=false","-Dhazelcast.socket.client.bind=false","-Dhazelcast.socket.client.bind.any=false","-Dhazelcast.logging.type=slf4j","-jar","app.jar"]

这是我使用的部署脚本,

    apiVersion: v1 # Kubernetes API version
    kind: Service # Kubernetes resource kind we are creating
    metadata: # Metadata of the resource kind we are creating
      name: spring-hazelcast-service
    spec:
      selector:
        app: spring-hazelcast-app
      ports:
        - protocol: "TCP"
          name: http-app
          port: 8081 # The port that the service is running on in the cluster
          targetPort: 8081 # The port exposed by the service
      type: LoadBalancer # type of the service. LoadBalancer indicates that our service will be external.
    ---
    apiVersion: apps/v1
    kind: Deployment # Kubernetes resource kind we are creating
    metadata:
      name: spring-hazelcast-app
    spec:
      selector:
        matchLabels:
          app: spring-hazelcast-app
      replicas: 1 # Number of replicas that will be created for this deployment
      template:
        metadata:
          labels:
            app: spring-hazelcast-app
        spec:
          containers:
            - name: hazelcast
              image: hazelcast/hazelcast:4.0.2
              workingDir: /opt
              ports:
                - name: hazelcast
                  containerPort: 5701
              env:
                - name: HZ_CLUSTERNAME
                  value: dev
                - name: JAVA_OPTS
                  value: -Dhazelcast.config=/opt/config/hazelcast.yml
              volumeMounts:
                - mountPath: "/opt/config/"
                  name: allconf
            - name: spring-hazelcast-app
              image: spring-hazelcast:1.0.3
              imagePullPolicy: Never #IfNotPresent
              ports:
                - containerPort: 8081 # The port that the container is running on in the cluster
          volumes:
            - name: allconf
              hostPath:
                path: /opt/config/   # directory location on host
                type: Directory # this field is optional
    ---
    apiVersion: v1 # Kubernetes API version
    kind: Service # Kubernetes resource kind we are creating
    metadata: # Metadata of the resource kind we are creating
      name: hazelcast-mc-service
    spec:
      selector:
        app: hazelcast-mc
      ports:
        - protocol: "TCP"
          name: mc-app
          port: 8080 # The port that the service is running on in the cluster
          targetPort: 8080 # The port exposed by the service
      type: LoadBalancer # type of the
      loadBalancerIP: "127.0.0.1"
    ---
    apiVersion: apps/v1
    kind: Deployment # Kubernetes resource kind we are creating
    metadata:
      name: hazelcast-mc
    spec:
      selector:
        matchLabels:
          app: hazelcast-mc
      replicas: 1 # Number of replicas that will be created for this deployment
      template:
        metadata:
          labels:
            app: hazelcast-mc
        spec:
          containers:
            - name: hazelcast-mc
              image: hazelcast/management-center
              ports:
                - containerPort: 8080 # The port that the container is running on in the cluster

这是我的应用程序日志,

      .   ____          _            __ _ _
     /\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
    ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
     \/  ___)| |_)| | | | | || (_| |  ) ) ) )
      '  |____| .__|_| |_|_| |_\__, | / / / /
     =========|_|==============|___/=/_/_/_/
     :: Spring Boot ::                (v2.5.4)
    
    2021-09-27 06:42:51.274  INFO 1 --- [           main] com.caching.Application                  : Starting Application using Java 11.0.5 on spring-hazelcast-app-7bdc8b7f7-bqdlt with PID 1 (/opt/app.jar started by root in /opt)
    2021-09-27 06:42:51.278  INFO 1 --- [           main] com.caching.Application                  : No active profile set, falling back to default profiles: default
    2021-09-27 06:42:55.986  INFO 1 --- [           main] c.h.c.impl.spi.ClientInvocationService   : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] Running with 2 response threads, dynamic=true
    2021-09-27 06:42:56.199  INFO 1 --- [           main] com.hazelcast.core.LifecycleService      : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] HazelcastClient 4.0.2 (20200702 - 2de3027) is STARTING
    2021-09-27 06:42:56.202  INFO 1 --- [           main] com.hazelcast.core.LifecycleService      : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] HazelcastClient 4.0.2 (20200702 - 2de3027) is STARTED
    WARNING: An illegal reflective access operation has occurred
    WARNING: Illegal reflective access by com.hazelcast.internal.networking.nio.SelectorOptimizer (jar:file:/opt/app.jar!/BOOT-INF/lib/hazelcast-all-4.0.2.jar!/) to field sun.nio.ch.SelectorImpl.selectedKeys
    WARNING: Please consider reporting this to the maintainers of com.hazelcast.internal.networking.nio.SelectorOptimizer
    WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
    WARNING: All illegal access operations will be denied in a future release
    2021-09-27 06:42:56.277  INFO 1 --- [           main] c.h.c.i.c.ClientConnectionManager        : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] Trying to connect to cluster: dev
    2021-09-27 06:42:56.302  INFO 1 --- [           main] c.h.c.i.c.ClientConnectionManager        : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] Trying to connect to [127.0.0.1]:5701
    2021-09-27 06:42:56.429  INFO 1 --- [           main] com.hazelcast.core.LifecycleService      : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] HazelcastClient 4.0.2 (20200702 - 2de3027) is CLIENT_CONNECTED
    2021-09-27 06:42:56.429  INFO 1 --- [           main] c.h.c.i.c.ClientConnectionManager        : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] Authenticated with server [172.17.0.3]:5701:c967f642-a7aa-4deb-a530-b56fb8f68c78, server version: 4.0.2, local address: /127.0.0.1:54373
    2021-09-27 06:42:56.436  INFO 1 --- [           main] c.h.internal.diagnostics.Diagnostics     : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
    2021-09-27 06:42:56.461  INFO 1 --- [21ad30a.event-4] c.h.c.impl.spi.ClientClusterService      : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] 
    
    Members [1] {
            Member [172.17.0.3]:5701 - c967f642-a7aa-4deb-a530-b56fb8f68c78
    }
    
    2021-09-27 06:42:56.803  INFO 1 --- [           main] c.h.c.i.s.ClientStatisticsService        : Client statistics is enabled with period 5 seconds.
    2021-09-27 06:42:57.878  INFO 1 --- [           main] c.h.i.config.AbstractConfigLocator       : Loading 'hazelcast.yaml' from the working directory.
    2021-09-27 06:42:57.934  WARN 1 --- [           main] c.h.i.impl.HazelcastInstanceFactory      : Hazelcast is starting in a Java modular environment (Java 9 and newer) but without proper access to required Java packages. Use additional Java arguments to provide Hazelcast access to Java internal API. The internal API access is used to get the best performance results. Arguments to be used:
     --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED
    2021-09-27 06:42:57.976  INFO 1 --- [           main] com.hazelcast.instance.AddressPicker     : [LOCAL] [dev] [4.0.2] Prefer IPv4 stack is true, prefer IPv6 addresses is false
    2021-09-27 06:42:57.987  INFO 1 --- [           main] com.hazelcast.instance.AddressPicker     : [LOCAL] [dev] [4.0.2] Picked [172.17.0.3]:5702, using socket ServerSocket[addr=/172.17.0.3,localport=5702], bind any local is false
    2021-09-27 06:42:58.004  INFO 1 --- [           main] com.hazelcast.system                     : [172.17.0.3]:5702 [dev] [4.0.2] Hazelcast 4.0.2 (20200702 - 2de3027) starting at [172.17.0.3]:5702
    2021-09-27 06:42:58.005  INFO 1 --- [           main] com.hazelcast.system                     : [172.17.0.3]:5702 [dev] [4.0.2] Copyright (c) 2008-2020, Hazelcast, Inc. All Rights Reserved.
    2021-09-27 06:42:58.047  INFO 1 --- [           main] c.h.s.i.o.impl.BackpressureRegulator     : [172.17.0.3]:5702 [dev] [4.0.2] Backpressure is disabled
    2021-09-27 06:42:58.373  INFO 1 --- [           main] com.hazelcast.instance.impl.Node         : [172.17.0.3]:5702 [dev] [4.0.2] Creating MulticastJoiner
    2021-09-27 06:42:58.380  WARN 1 --- [           main] com.hazelcast.cp.CPSubsystem             : [172.17.0.3]:5702 [dev] [4.0.2] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
    2021-09-27 06:42:58.676  INFO 1 --- [           main] c.h.s.i.o.impl.OperationExecutorImpl     : [172.17.0.3]:5702 [dev] [4.0.2] Starting 2 partition threads and 3 generic threads (1 dedicated for priority tasks)
    2021-09-27 06:42:58.682  INFO 1 --- [           main] c.h.internal.diagnostics.Diagnostics     : [172.17.0.3]:5702 [dev] [4.0.2] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
    
    
    2021-09-27 06:42:58.687  INFO 1 --- [           main] com.hazelcast.core.LifecycleService      : [172.17.0.3]:5702 [dev] [4.0.2] [172.17.0.3]:5702 is STARTING
    2021-09-27 06:42:58.923  INFO 1 --- [           main] c.h.i.cluster.impl.MulticastJoiner       : [172.17.0.3]:5702 [dev] [4.0.2] Trying to join to discovered node: [172.17.0.3]:5701
    2021-09-27 06:42:58.932  INFO 1 --- [cached.thread-3] c.h.internal.nio.tcp.TcpIpConnector      : [172.17.0.3]:5702 [dev] [4.0.2] Connecting to /172.17.0.3:5701, timeout: 10000, bind-any: false
    2021-09-27 06:42:58.955  INFO 1 --- [.IO.thread-in-0] c.h.internal.nio.tcp.TcpIpConnection     : [172.17.0.3]:5702 [dev] [4.0.2] Initialized new cluster connection between /172.17.0.3:40242 and /172.17.0.3:5701
    2021-09-27 06:43:04.948  INFO 1 --- [21ad30a.event-3] c.h.c.impl.spi.ClientClusterService      : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] 
    
    Members [2] {
            Member [172.17.0.3]:5701 - c967f642-a7aa-4deb-a530-b56fb8f68c78
            Member [172.17.0.3]:5702 - 08dfe633-46b2-4581-94c7-81b6d0bc3ce3
    }
    
    2021-09-27 06:43:04.959  WARN 1 --- [ration.thread-0] c.h.c.i.operation.OnJoinCacheOperation   : [172.17.0.3]:5702 [dev] [4.0.2] This member is joining a cluster whose members support JCache, however the cache-api artifact is missing from this member's classpath. In case JCache API will be used, add cache-api artifact in this member's classpath and restart the member.
    2021-09-27 06:43:04.963  INFO 1 --- [ration.thread-0] c.h.internal.cluster.ClusterService      : [172.17.0.3]:5702 [dev] [4.0.2] 
    
    Members {size:2, ver:2} [
            Member [172.17.0.3]:5701 - c967f642-a7aa-4deb-a530-b56fb8f68c78
            Member [172.17.0.3]:5702 - 08dfe633-46b2-4581-94c7-81b6d0bc3ce3 this
    ]
    
    2021-09-27 06:43:05.466  INFO 1 --- [ration.thread-1] c.h.c.i.p.t.AuthenticationMessageTask    : [172.17.0.3]:5702 [dev] [4.0.2] Received auth from Connection[id=2, /172.17.0.3:5702->/172.17.0.3:40773, qualifier=null, endpoint=[172.17.0.3]:40773, alive=true, connectionType=JVM], successfully authenticated, clientUuid: 8843f057-c856-4739-80ae-4bc930559bd5, client version: 4.0.2
    2021-09-27 06:43:05.468  INFO 1 --- [d30a.internal-3] c.h.c.i.c.ClientConnectionManager        : b1bdd9bb-2879-4161-95fd-2b6e321ad30a [dev] [4.0.2] Authenticated with server [172.17.0.3]:5702:08dfe633-46b2-4581-94c7-81b6d0bc3ce3, server version: 4.0.2, local address: /172.17.0.3:40773
    2021-09-27 06:43:05.968  INFO 1 --- [           main] com.hazelcast.core.LifecycleService      : [172.17.0.3]:5702 [dev] [4.0.2] [172.17.0.3]:5702 is STARTED
    2021-09-27 06:43:06.237  INFO 1 --- [           main] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8081
    2021-09-27 06:43:06.251  INFO 1 --- [           main] com.caching.Application                  : Started Application in 17.32 seconds (JVM running for 21.02)

这里是Hazelcast管理中心会员名单,

最后我的问题是,

  1. 为什么我看到 2 个成员,其中只部署了一个 sidecar 缓存容器?
  2. 我需要做哪些修改才能达到最初的目标?

根据Spring Boot documentation for Hazelcast feature

If a client can’t be created, Spring Boot attempts to configure an embedded server.

Spring Boot 从应用程序容器 hazelcast.yaml 启动嵌入式服务器,并使用多播加入 Hazelcast 容器。

您应该将 Spring 启动应用程序容器中的 hazelcast.yaml 替换为 hazelcast-client.yaml,内容如下:

hazelcast-client:
  cluster-name: "dev"
  network:
    cluster-members:
    - "127.0.0.1:5701"

完成后 Spring 引导将自动配置客户端 HazelcastInstance bean,您将能够像这样更改缓存客户端:

@Component
public class CacheClient {
    
    private static final String ITEMS = "items";
    
    private final HazelcastInstance client;

    public CacheClient(HazelcastInstance client) {
        this.client = client;
    }

    public Item put(String number, Item item){
        IMap<String, Item> map = client.getMap(ITEMS);
        return map.putIfAbsent(number, item);
    }

    public Item get(String key){
        IMap<String, Item> map = client.getMap(ITEMS);
        return map.get(key);
    }
}