Spring 集成、kinesis 活页夹、localstack 奇怪的行为

Question

我在使用来自 testcontainers 的 spring-integration、kinesis binder 和 localstack（aws 本地环境）时遇到了非常奇怪的问题。我有 2 个频道——A 和 B。在每个频道上我都有生产者和消费者。我的测试非常简单——我正在发送消息，并验证消费者收到的消息。所以我为每个频道写了 1 个测试，它对我有用。第三次测试使用通道 A 它不起作用，因为消费者没有收到消息。

这是第三次测试的日志

2020-05-14 17:27:40.654  INFO 29076 --- [           main] com.blabla.kinesis.demo.MessageProducer  : Sending StatusEvent(status=message2)
2020-05-14 17:27:40.663 DEBUG 29076 --- [           main] o.s.i.a.outbound.KinesisMessageHandler   : org.springframework.integration.aws.outbound.KinesisMessageHandler@5895c065 received message: GenericMessage [payload=byte[57], headers={contentType=application/json, id=de100ebb-518c-2025-7d71-99c94cd8207c, timestamp=1589466460663}]
2020-05-14 17:27:41.455  INFO 29076 --- [esis-consumer-1] a.i.k.KinesisMessageDrivenChannelAdapter : The [ShardConsumer{shardOffset=KinesisShardOffset{iteratorType=LATEST, sequenceNumber='null', timestamp=null, stream='input', shard='shardId-000000000000', reset=false}, state=NEW}] has been started.
2020-05-14 17:27:42.661 DEBUG 29076 --- [esis-consumer-1] a.i.k.KinesisMessageDrivenChannelAdapter : No records for [ShardConsumer{shardOffset=KinesisShardOffset{iteratorType=LATEST, sequenceNumber='null', timestamp=null, stream='input', shard='shardId-000000000000', reset=false}, state=CONSUME}] on sequenceNumber [null]. Suspend consuming for [1000] milliseconds.

从日志中我看到生产者发送了消息，但是消费者在它之后启动了。如果我之前发送了另一条消息，第二条消息没有任何问题，第一条消失了

我的pom.xml:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.2.6.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.blabla.kinesis</groupId>
    <artifactId>demo</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>demo</name>
    <description>Demo project for Spring Boot</description>

    <properties>
        <java.version>11</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-aop</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
        <dependency>
            <groupId>org.springframework.cloud</groupId>
            <artifactId>spring-cloud-stream-binder-kinesis</artifactId>
            <version>2.0.1.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>org.testcontainers</groupId>
            <artifactId>localstack</artifactId>
            <version>1.14.1</version>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <optional>true</optional>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
            <exclusions>
                <exclusion>
                    <groupId>org.junit.vintage</groupId>
                    <artifactId>junit-vintage-engine</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
    </dependencies>
    <dependencyManagement>

        <dependencies>
            <dependency>
                <groupId>com.amazonaws</groupId>
                <artifactId>aws-java-sdk-core</artifactId>
                <version>1.11.415</version>
            </dependency>
        </dependencies>
    </dependencyManagement>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>

</project>

Answer 1

经过几个小时的调查，我发现了问题所在。根本原因是在我的配置中我只定义了 "sync" 客户端，如下所示：

@Autowired
private LocalStackContainer localStackContainer;

@Bean
public AmazonKinesis kinesisClient() {
    return AmazonKinesisClientBuilder.standard()
            .withEndpointConfiguration(localStackContainer.getEndpointConfiguration(KINESIS))
            .build();
}

@Bean
public AmazonDynamoDB dynamoDBClient() {
    return  AmazonDynamoDBClientBuilder.standard()
            .withEndpointConfiguration(localStackContainer.getEndpointConfiguration(DYNAMODB))
            .build();
}

打开所有调试日志后，我注意到，Spring 使用 AmazonKinesisAsyncClient 对 "real" AWS 进行了 api 次调用。以下配置解决了问题：

@Bean
public AmazonDynamoDB dynamoDBClientAsync() {
    return  AmazonDynamoDBAsyncClientBuilder.standard()
            .withEndpointConfiguration(localStackContainer.getEndpointConfiguration(DYNAMODB))
            .build();
}
@Bean
public AmazonKinesis kinesisClientAsync() {
    return AmazonKinesisAsyncClientBuilder.standard()
            .withEndpointConfiguration(localStackContainer.getEndpointConfiguration(KINESIS))
            .build();
}

Spring 集成、kinesis 活页夹、localstack 奇怪的行为

Spring integration, kinesis binder, localstack strange behaviour

spring-integration

amazon-kinesis

localstack