Failing in MiNiFi-Tutorial, toolkit error: Connection with id ... has invalid destination id
Failing in MiNiFi-Tutorial, toolkit error: Connection with id ... has invalid destination id
我遵循了这个教程：Getting started with MiNiFi
因此，我的根进程组中有一个input port，还有一个用于MiNiFi的进程组。 MiNiFi Process Group，包含一个Tail File Processor，一个Remote Process Group和它们之间的连接。
在 MiNiFi Process Group 中我正在保存模板，之后我从模板列表下载它并 运行 命令：
minifi-toolkit-0.5.0\bin\config.bat transform minifi\MiNiFi2.xml minifi\minifi_config\config.yml
但它失败了并说：
The JAVA_HOME environment variable is not defined correctly.
Instead the PATH will be used to find the java executable.
There are validation errors with the template, still outputting YAML but it will need to be edited.
Connection with id 10de7342-41ef-37a5-0000-000000000000 has invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
我重新创建了所有内容，但问题仍然存在。
这是我的模板：
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.3">
<description></description>
<groupId>db4ce9c5-016e-1000-bce2-b531930a6edd</groupId>
<name>MiNiFi</name>
<snippet>
<connections>
<id>10de7342-41ef-37a5-0000-000000000000</id>
<parentGroupId>e544a343-e2f0-3112-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>1e18bdb9-7df3-3fba-0000-000000000000</groupId>
<id>9b02eb45-3c2d-3fb4-91cf-880d5038c0cd</id>
<type>REMOTE_INPUT_PORT</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>e544a343-e2f0-3112-0000-000000000000</groupId>
<id>246f7d78-f8e1-3c88-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<processors>
<id>246f7d78-f8e1-3c88-0000-000000000000</id>
<parentGroupId>e544a343-e2f0-3112-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>16.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>tail-mode</key>
<value>
<name>tail-mode</name>
</value>
</entry>
<entry>
<key>File to Tail</key>
<value>
<name>File to Tail</name>
</value>
</entry>
<entry>
<key>Rolling Filename Pattern</key>
<value>
<name>Rolling Filename Pattern</name>
</value>
</entry>
<entry>
<key>tail-base-directory</key>
<value>
<name>tail-base-directory</name>
</value>
</entry>
<entry>
<key>Initial Start Position</key>
<value>
<name>Initial Start Position</name>
</value>
</entry>
<entry>
<key>File Location</key>
<value>
<name>File Location</name>
</value>
</entry>
<entry>
<key>tailfile-recursive-lookup</key>
<value>
<name>tailfile-recursive-lookup</name>
</value>
</entry>
<entry>
<key>tailfile-lookup-frequency</key>
<value>
<name>tailfile-lookup-frequency</name>
</value>
</entry>
<entry>
<key>tailfile-maximum-age</key>
<value>
<name>tailfile-maximum-age</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>tail-mode</key>
<value>Single file</value>
</entry>
<entry>
<key>File to Tail</key>
<value>/opt/minifi/minifi_test_dir</value>
</entry>
<entry>
<key>Rolling Filename Pattern</key>
</entry>
<entry>
<key>tail-base-directory</key>
</entry>
<entry>
<key>Initial Start Position</key>
<value>Beginning of File</value>
</entry>
<entry>
<key>File Location</key>
<value>Local</value>
</entry>
<entry>
<key>tailfile-recursive-lookup</key>
<value>false</value>
</entry>
<entry>
<key>tailfile-lookup-frequency</key>
<value>10 minutes</value>
</entry>
<entry>
<key>tailfile-maximum-age</key>
<value>24 hours</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>TailFile</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.TailFile</type>
</processors>
<remoteProcessGroups>
<id>1e18bdb9-7df3-3fba-0000-000000000000</id>
<parentGroupId>e544a343-e2f0-3112-0000-000000000000</parentGroupId>
<position>
<x>696.0</x>
<y>0.0</y>
</position>
<comments></comments>
<communicationsTimeout>30 sec</communicationsTimeout>
<contents>
<inputPorts>
<batchSettings/>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<connected>true</connected>
<exists>true</exists>
<groupId>db4d2e20-016e-1000-d3c7-73a67e2b1d67</groupId>
<id>9b02eb45-3c2d-3fb4-91cf-880d5038c0cd</id>
<name>input_minifi</name>
<targetId>dadd7f29-016e-1000-3c91-89495c813674</targetId>
<targetRunning>false</targetRunning>
<transmitting>false</transmitting>
<useCompression>false</useCompression>
</inputPorts>
</contents>
<proxyHost></proxyHost>
<proxyUser></proxyUser>
<targetUri>http://nifi-server:8080/nifi</targetUri>
<targetUris>http://nifi-server:8080/nifi</targetUris>
<transportProtocol>RAW</transportProtocol>
<yieldDuration>10 sec</yieldDuration>
</remoteProcessGroups>
</snippet>
<timestamp>12/06/2019 13:04:03 UTC</timestamp>
</template>
这是工具包结果：
MiNiFi Config Version: 3
Flow Controller:
name: MiNiFi
comment: ''
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
variable registry properties: ''
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Provenance Repository:
provenance rollover time: 1 min
implementation: org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: ''
keystore type: ''
keystore password: ''
key password: ''
truststore: ''
truststore type: ''
truststore password: ''
ssl protocol: ''
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors:
- id: 246f7d78-f8e1-3c88-0000-000000000000
name: TailFile
class: org.apache.nifi.processors.standard.TailFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list: []
Properties:
File Location: Local
File to Tail: /opt/minifi/minifi_test_dir
Initial Start Position: Beginning of File
Rolling Filename Pattern:
tail-base-directory:
tail-mode: Single file
tailfile-lookup-frequency: 10 minutes
tailfile-maximum-age: 24 hours
tailfile-recursive-lookup: 'false'
Controller Services: []
Process Groups: []
Input Ports: []
Output Ports: []
Funnels: []
Connections:
- id: 10de7342-41ef-37a5-0000-000000000000
name: TailFile/success/9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
source id: 246f7d78-f8e1-3c88-0000-000000000000
source relationship names:
- success
destination id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
max work queue size: 10000
max work queue data size: 1 GB
flowfile expiration: 0 sec
queue prioritizer class: ''
Remote Process Groups:
- id: 1e18bdb9-7df3-3fba-0000-000000000000
name: ''
url: http://nifi-server:8080/nifi
comment: ''
timeout: 30 sec
yield period: 10 sec
transport protocol: RAW
proxy host: ''
proxy port: ''
proxy user: ''
proxy password: ''
local network interface: ''
Input Ports:
- id: dadd7f29-016e-1000-3c91-89495c813674
name: input_minifi
comment: ''
max concurrent tasks: 1
use compression: false
Output Ports: []
NiFi Properties Overrides: {}
编辑：
工具包中似乎存在错误（MINIFI-521 由@seb 报告），其中 targetId 在模板中用作输入端口 ID，但连接 ID 未正确映射。
上一个：
如果我没记错的话，该工具包希望它处理的模板是完全独立的。
模板正在引用根进程组的输入端口。
当只导出模板并转换为与 MiNiFi 一起使用时，这将导致它丢失。
我建议将您的输入端口向下移动到您要为其创建模板的进程组中。
我遵循了这个教程：Getting started with MiNiFi
因此，我的根进程组中有一个input port，还有一个用于MiNiFi的进程组。 MiNiFi Process Group，包含一个Tail File Processor，一个Remote Process Group和它们之间的连接。
在 MiNiFi Process Group 中我正在保存模板，之后我从模板列表下载它并 运行 命令：
minifi-toolkit-0.5.0\bin\config.bat transform minifi\MiNiFi2.xml minifi\minifi_config\config.yml
但它失败了并说：
The JAVA_HOME environment variable is not defined correctly.
Instead the PATH will be used to find the java executable.
There are validation errors with the template, still outputting YAML but it will need to be edited.
Connection with id 10de7342-41ef-37a5-0000-000000000000 has invalid destination id 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
我重新创建了所有内容，但问题仍然存在。
这是我的模板：
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<template encoding-version="1.3">
<description></description>
<groupId>db4ce9c5-016e-1000-bce2-b531930a6edd</groupId>
<name>MiNiFi</name>
<snippet>
<connections>
<id>10de7342-41ef-37a5-0000-000000000000</id>
<parentGroupId>e544a343-e2f0-3112-0000-000000000000</parentGroupId>
<backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold>
<backPressureObjectThreshold>10000</backPressureObjectThreshold>
<destination>
<groupId>1e18bdb9-7df3-3fba-0000-000000000000</groupId>
<id>9b02eb45-3c2d-3fb4-91cf-880d5038c0cd</id>
<type>REMOTE_INPUT_PORT</type>
</destination>
<flowFileExpiration>0 sec</flowFileExpiration>
<labelIndex>1</labelIndex>
<loadBalanceCompression>DO_NOT_COMPRESS</loadBalanceCompression>
<loadBalancePartitionAttribute></loadBalancePartitionAttribute>
<loadBalanceStatus>LOAD_BALANCE_NOT_CONFIGURED</loadBalanceStatus>
<loadBalanceStrategy>DO_NOT_LOAD_BALANCE</loadBalanceStrategy>
<name></name>
<selectedRelationships>success</selectedRelationships>
<source>
<groupId>e544a343-e2f0-3112-0000-000000000000</groupId>
<id>246f7d78-f8e1-3c88-0000-000000000000</id>
<type>PROCESSOR</type>
</source>
<zIndex>0</zIndex>
</connections>
<processors>
<id>246f7d78-f8e1-3c88-0000-000000000000</id>
<parentGroupId>e544a343-e2f0-3112-0000-000000000000</parentGroupId>
<position>
<x>0.0</x>
<y>16.0</y>
</position>
<bundle>
<artifact>nifi-standard-nar</artifact>
<group>org.apache.nifi</group>
<version>1.10.0</version>
</bundle>
<config>
<bulletinLevel>WARN</bulletinLevel>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<descriptors>
<entry>
<key>tail-mode</key>
<value>
<name>tail-mode</name>
</value>
</entry>
<entry>
<key>File to Tail</key>
<value>
<name>File to Tail</name>
</value>
</entry>
<entry>
<key>Rolling Filename Pattern</key>
<value>
<name>Rolling Filename Pattern</name>
</value>
</entry>
<entry>
<key>tail-base-directory</key>
<value>
<name>tail-base-directory</name>
</value>
</entry>
<entry>
<key>Initial Start Position</key>
<value>
<name>Initial Start Position</name>
</value>
</entry>
<entry>
<key>File Location</key>
<value>
<name>File Location</name>
</value>
</entry>
<entry>
<key>tailfile-recursive-lookup</key>
<value>
<name>tailfile-recursive-lookup</name>
</value>
</entry>
<entry>
<key>tailfile-lookup-frequency</key>
<value>
<name>tailfile-lookup-frequency</name>
</value>
</entry>
<entry>
<key>tailfile-maximum-age</key>
<value>
<name>tailfile-maximum-age</name>
</value>
</entry>
</descriptors>
<executionNode>ALL</executionNode>
<lossTolerant>false</lossTolerant>
<penaltyDuration>30 sec</penaltyDuration>
<properties>
<entry>
<key>tail-mode</key>
<value>Single file</value>
</entry>
<entry>
<key>File to Tail</key>
<value>/opt/minifi/minifi_test_dir</value>
</entry>
<entry>
<key>Rolling Filename Pattern</key>
</entry>
<entry>
<key>tail-base-directory</key>
</entry>
<entry>
<key>Initial Start Position</key>
<value>Beginning of File</value>
</entry>
<entry>
<key>File Location</key>
<value>Local</value>
</entry>
<entry>
<key>tailfile-recursive-lookup</key>
<value>false</value>
</entry>
<entry>
<key>tailfile-lookup-frequency</key>
<value>10 minutes</value>
</entry>
<entry>
<key>tailfile-maximum-age</key>
<value>24 hours</value>
</entry>
</properties>
<runDurationMillis>0</runDurationMillis>
<schedulingPeriod>0 sec</schedulingPeriod>
<schedulingStrategy>TIMER_DRIVEN</schedulingStrategy>
<yieldDuration>1 sec</yieldDuration>
</config>
<executionNodeRestricted>false</executionNodeRestricted>
<name>TailFile</name>
<relationships>
<autoTerminate>false</autoTerminate>
<name>success</name>
</relationships>
<state>STOPPED</state>
<style/>
<type>org.apache.nifi.processors.standard.TailFile</type>
</processors>
<remoteProcessGroups>
<id>1e18bdb9-7df3-3fba-0000-000000000000</id>
<parentGroupId>e544a343-e2f0-3112-0000-000000000000</parentGroupId>
<position>
<x>696.0</x>
<y>0.0</y>
</position>
<comments></comments>
<communicationsTimeout>30 sec</communicationsTimeout>
<contents>
<inputPorts>
<batchSettings/>
<comments></comments>
<concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount>
<connected>true</connected>
<exists>true</exists>
<groupId>db4d2e20-016e-1000-d3c7-73a67e2b1d67</groupId>
<id>9b02eb45-3c2d-3fb4-91cf-880d5038c0cd</id>
<name>input_minifi</name>
<targetId>dadd7f29-016e-1000-3c91-89495c813674</targetId>
<targetRunning>false</targetRunning>
<transmitting>false</transmitting>
<useCompression>false</useCompression>
</inputPorts>
</contents>
<proxyHost></proxyHost>
<proxyUser></proxyUser>
<targetUri>http://nifi-server:8080/nifi</targetUri>
<targetUris>http://nifi-server:8080/nifi</targetUris>
<transportProtocol>RAW</transportProtocol>
<yieldDuration>10 sec</yieldDuration>
</remoteProcessGroups>
</snippet>
<timestamp>12/06/2019 13:04:03 UTC</timestamp>
</template>
这是工具包结果：
MiNiFi Config Version: 3
Flow Controller:
name: MiNiFi
comment: ''
Core Properties:
flow controller graceful shutdown period: 10 sec
flow service write delay interval: 500 ms
administrative yield duration: 30 sec
bored yield duration: 10 millis
max concurrent threads: 1
variable registry properties: ''
FlowFile Repository:
partitions: 256
checkpoint interval: 2 mins
always sync: false
Swap:
threshold: 20000
in period: 5 sec
in threads: 1
out period: 5 sec
out threads: 4
Content Repository:
content claim max appendable size: 10 MB
content claim max flow files: 100
always sync: false
Provenance Repository:
provenance rollover time: 1 min
implementation: org.apache.nifi.provenance.MiNiFiPersistentProvenanceRepository
Component Status Repository:
buffer size: 1440
snapshot frequency: 1 min
Security Properties:
keystore: ''
keystore type: ''
keystore password: ''
key password: ''
truststore: ''
truststore type: ''
truststore password: ''
ssl protocol: ''
Sensitive Props:
key:
algorithm: PBEWITHMD5AND256BITAES-CBC-OPENSSL
provider: BC
Processors:
- id: 246f7d78-f8e1-3c88-0000-000000000000
name: TailFile
class: org.apache.nifi.processors.standard.TailFile
max concurrent tasks: 1
scheduling strategy: TIMER_DRIVEN
scheduling period: 0 sec
penalization period: 30 sec
yield period: 1 sec
run duration nanos: 0
auto-terminated relationships list: []
Properties:
File Location: Local
File to Tail: /opt/minifi/minifi_test_dir
Initial Start Position: Beginning of File
Rolling Filename Pattern:
tail-base-directory:
tail-mode: Single file
tailfile-lookup-frequency: 10 minutes
tailfile-maximum-age: 24 hours
tailfile-recursive-lookup: 'false'
Controller Services: []
Process Groups: []
Input Ports: []
Output Ports: []
Funnels: []
Connections:
- id: 10de7342-41ef-37a5-0000-000000000000
name: TailFile/success/9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
source id: 246f7d78-f8e1-3c88-0000-000000000000
source relationship names:
- success
destination id: 9b02eb45-3c2d-3fb4-91cf-880d5038c0cd
max work queue size: 10000
max work queue data size: 1 GB
flowfile expiration: 0 sec
queue prioritizer class: ''
Remote Process Groups:
- id: 1e18bdb9-7df3-3fba-0000-000000000000
name: ''
url: http://nifi-server:8080/nifi
comment: ''
timeout: 30 sec
yield period: 10 sec
transport protocol: RAW
proxy host: ''
proxy port: ''
proxy user: ''
proxy password: ''
local network interface: ''
Input Ports:
- id: dadd7f29-016e-1000-3c91-89495c813674
name: input_minifi
comment: ''
max concurrent tasks: 1
use compression: false
Output Ports: []
NiFi Properties Overrides: {}
编辑：
工具包中似乎存在错误（MINIFI-521 由@seb 报告），其中 targetId 在模板中用作输入端口 ID，但连接 ID 未正确映射。
上一个：
如果我没记错的话，该工具包希望它处理的模板是完全独立的。
模板正在引用根进程组的输入端口。
当只导出模板并转换为与 MiNiFi 一起使用时，这将导致它丢失。
我建议将您的输入端口向下移动到您要为其创建模板的进程组中。