StreamSets 处理器找不到 Redis 库

Question

我有两个装有 Redis 和 StreamSet 的容器。我想在 Java 中编写自定义处理器并放入管道。但是，当我将教程中的代码添加到处理器、将 jar 发送到 lib 并尝试启动时，我遇到了异常。请问你能帮帮我吗？

处理器教程 - https://github.com/streamsets/tutorials/blob/master/tutorial-processor/readme.md

Redis 教程 - https://dzone.com/articles/quickstart-how-to-use-redis-on-java

  redis:
    image: redis
    ports:
      - 6379:6379
    command: redis-server


  streamsets:
    build:
      context: "."
      args:
        SDC_LIBS: streamsets-datacollector-apache-kafka_2_0-lib                ports:
      - 18630:18630
    volumes:
      - ./user-libs:/opt/streamsets-datacollector-3.9.0/user-libs

样本处理器中的启动：

  @Override
  protected List<ConfigIssue> init() {
      ....
      Config config = new Config();
      config.useSingleServer().setAddress("redis://redis:6379");
      redisson = Redisson.create(config);
      ... }



  <dependency>
      <groupId>org.redisson</groupId>
      <artifactId>redisson</artifactId>
      <version>3.10.4</version>
      <scope>provided</scope>
  </dependency>

处理器代码测试成功。

日志

streamsets_1  | 2020-03-21 14:00:19,954 [user:] [pipeline:] [runner:] [thread:main] [stage:] ERROR Main - Abnormal exit: java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;

streamsets_1  | java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;
streamsets_1  |         at java.lang.Class.getDeclaredFields0(Native Method)
streamsets_1  |         at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2614)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2629)
streamsets_1  |         at java.lang.Class.getFields(Class.java:1557)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:82)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:65)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.validate(StageDefinitionExtractor.java:141)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.extract(StageDefinitionExtractor.java:197)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.loadStages(ClassLoaderStageLibraryTask.java:557)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.initTask(ClassLoaderStageLibraryTask.java:278)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.CompositeTask.initTask(CompositeTask.java:44)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.TaskWrapper.init(TaskWrapper.java:40)
streamsets_1  |         at com.streamsets.datacollector.main.Main.lambda$doMain(Main.java:123)
streamsets_1  |         at java.security.AccessController.doPrivileged(Native Method)
streamsets_1  |         at com.streamsets.datacollector.security.SecurityUtil.doAs(SecurityUtil.java:92)
streamsets_1  |         at com.streamsets.datacollector.main.Main.doMain(Main.java:158)
streamsets_1  |         at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
streamsets_1  |         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
streamsets_1  |         at java.lang.reflect.Method.invoke(Method.java:498)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
streamsets_1  | Caused by: java.lang.ClassNotFoundException: org.redisson.api.RedissonClient
streamsets_1  |         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
streamsets_1  |         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         ... 26 more
streamsets_1  |
streamsets_1  | Abnormal exit: java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;Check STDERR for more details
streamsets_1  |
streamsets_1  | java.lang.NoClassDefFoundError: Lorg/redisson/api/RedissonClient;
streamsets_1  |         at java.lang.Class.getDeclaredFields0(Native Method)
streamsets_1  |         at java.lang.Class.privateGetDeclaredFields(Class.java:2583)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2614)
streamsets_1  |         at java.lang.Class.privateGetPublicFields(Class.java:2629)
streamsets_1  |         at java.lang.Class.getFields(Class.java:1557)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:82)
streamsets_1  |         at com.streamsets.datacollector.definition.ConfigDefinitionExtractor.validate(ConfigDefinitionExtractor.java:65)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.validate(StageDefinitionExtractor.java:141)
streamsets_1  |         at com.streamsets.datacollector.definition.StageDefinitionExtractor.extract(StageDefinitionExtractor.java:197)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.loadStages(ClassLoaderStageLibraryTask.java:557)
streamsets_1  |         at com.streamsets.datacollector.stagelibrary.ClassLoaderStageLibraryTask.initTask(ClassLoaderStageLibraryTask.java:278)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.CompositeTask.initTask(CompositeTask.java:44)
streamsets_1  |         at com.streamsets.datacollector.task.AbstractTask.init(AbstractTask.java:62)
streamsets_1  |         at com.streamsets.datacollector.task.TaskWrapper.init(TaskWrapper.java:40)
streamsets_1  |         at com.streamsets.datacollector.main.Main.lambda$doMain(Main.java:123)
streamsets_1  |         at java.security.AccessController.doPrivileged(Native Method)
streamsets_1  |         at com.streamsets.datacollector.security.SecurityUtil.doAs(SecurityUtil.java:92)
streamsets_1  |         at com.streamsets.datacollector.main.Main.doMain(Main.java:158)
streamsets_1  |         at com.streamsets.datacollector.main.DataCollectorMain.main(DataCollectorMain.java:53)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
streamsets_1  |         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
streamsets_1  |         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
streamsets_1  |         at java.lang.reflect.Method.invoke(Method.java:498)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.bootstrap(BootstrapMain.java:351)
streamsets_1  |         at com.streamsets.pipeline.BootstrapMain.main(BootstrapMain.java:110)
streamsets_1  | Caused by: java.lang.ClassNotFoundException: org.redisson.api.RedissonClient
streamsets_1  |         at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
streamsets_1  |         at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
streamsets_1  |         at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:353)
streamsets_1  |         at com.streamsets.pipeline.SDCClassLoader.loadClass(SDCClassLoader.java:316)
streamsets_1  |         ... 26 more

Answer 1

问题可能出在您的 dependency scope 上。您已指定 provided，其定义为：

This is much like compile, but indicates you expect the JDK or a container to provide the dependency at runtime. For example, when building a web application for the Java Enterprise Edition, you would set the dependency on the Servlet API and related Java EE APIs to scope provided because the web container provides those classes. This scope is only available on the compilation and test classpath, and is not transitive.

您应该指定 compile 范围。

StreamSets 处理器找不到 Redis 库

StreamSets processor cant find the Redis libs

java

redis

docker

docker-compose

streamsets