如何调试为什么 Telegraf 没有将数据从 MQTT 发送到 influxdb
How to debug why Telegraf is not sending data from MQTT to influxdb
我正尝试按如下方式设置 TIG 堆栈：
传感器数据 -> MQTT (Eclipse-Mosquitto) -> Telegraf -> InfluxDB -> Grafana
我还没有设置传感器数据的来源，但在目前的状态下，一切都在编译并且 运行s 没有明显的错误，而且似乎都已连接但发布到 MQTT 主题的数据是没有出现在我的 Influxdb 数据库中。
$ sudo docker-compose up
Creating network "tig-samson-feature-setup-docker-compose_default" with the default driver
Creating mosquitto ... done
Creating influxdb ... done
Creating grafana ... done
Creating telegraf ... done
Attaching to influxdb, mosquitto, grafana, telegraf
mosquitto | 1583407265: mosquitto version 1.6.9 starting
mosquitto | 1583407265: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto | 1583407265: Opening ipv4 listen socket on port 1883.
mosquitto | 1583407265: Opening ipv6 listen socket on port 1883.
mosquitto | 1583407266: New connection from 192.168.32.1 on port 1883.
mosquitto | 1583407266: New client connected from 192.168.32.1 as mosqsub|5779-jackbridge (p1, c1, k60).
grafana | t=2020-03-05T11:21:05+0000 lvl=info msg="Starting Grafana" logger=server version=6.6.2 commit=3fa63cfc34 branch=HEAD compiled=2020-02-20T12:03:49+0000
grafana | t=2020-03-05T11:21:05+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
grafana | t=2020-03-05T11:21:05+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
grafana | t=2020-03-05T11:21:05+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
...continues indefinitely.. can add more detail if it's helpfuL!
下面是我的一些代码和我尝试过的一些东西
文件架构 运行在 ubuntu 虚拟机上（运行在 mac 上的 virtualbox 中）
docker-compose.yml
telegraf.conf
influxdb (folder)
-----data (folder)
-----------data (locked folder auto generated)
-----------meta (locked folder auto generated)
-----------wal (locked folder auto generated)
data
----mosquitto (empty folder)
docker-撰写
# # docker-compose.yml
version: '3'
services:
influxdb:
container_name: influxdb
image: influxdb
ports:
- "8086:8086"
environment:
- INFLUXDB_DB=sensors
- INFLUXDB_ADMIN_USER=admin
- INFLUXDB_ADMIN_PASSWORD=telegraf
volumes:
- ./influxdb/data:/var/lib/influxdb
restart: always
telegraf:
container_name: telegraf
image: telegraf
restart: always
depends_on:
- mosquitto
- influxdb
volumes:
- ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
grafana:
container_name: grafana
image: grafana/grafana
links:
- influxdb
hostname: grafana
ports:
- "3000:3000"
mosquitto:
container_name: mosquitto
image: eclipse-mosquitto
ports:
- "1883:1883"
- "9001:9001"
restart: always
volumes:
- ./mosquitto.conf:/mqtt/config:ro
# - /data/mosquitto:mosquitto/data
（我用docker-compose up启动程序）
telegraf.conf
[[outputs.file]]
# ## Files to write to, "stdout" is a specially handled file.
files = ["stdout"]
# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
servers = ["tcp://localhost:1883"]
topics = [
"sensors"
]
data_format = "influx"
# ## MQTT broker URLs to be used. The format should be scheme://host:port,
# ## schema can be tcp, ssl, or ws.
# servers = ["tcp://127.0.0.1:1883"]
[[outputs.influxdb]]
urls = ["http://influxdb:8086"]
database = "sensors"
skip_database_creation = true
username = "telegraf"
password = "telegraf"
我可以访问 telegraf docker 容器并 ping influxdb 所以它似乎已连接
root@fc1d9af8f0c6:/# curl -sl -I http://influxdb:8086/ping
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 12b4923a-5ec7-11ea-8006-0242c0a81002
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.10
X-Request-Id: 12b4923a-5ec7-11ea-8006-0242c0a81002
Date: Thu, 05 Mar 2020 09:52:49 GMT
我可以在其他终端发布订阅MQTT：
$ mosquitto_pub -t 'sensors' -m 'strength, site=room1 value=HELLO tony'
这是在订阅传感器的另一个终端中接收到的
$ mosquitto_sub -t 'sensors' -v # note this is executed before mosquitto_pub
sensors strength, site=room1 value=HELLO tony
当我执行 运行 mosquitto_sub 时，终端 运行ning docker-compose.activity 中没有 activity。
当我 运行 mosquitto_pub 客户端被检测为正在连接但发布事件似乎没有注册
mosquitto | 1583407850: New connection from 192.168.32.1 on port 1883.
mosquitto | 1583407850: New client connected from 192.168.32.1 as mosqpub|7039-jackbridge (p1, c1, k60).
mosquitto | 1583407850: Client mosqpub|7039-jackbridge disconnected.
Telegraf 似乎正在向 Influxdb 发送指标。在 telegraf [[outputs.file]] 中，我正在发送上面的文件 =['stdout']。目前尚不清楚指标是否到达 influxdb，因为我无法找到它们。
终端输出示例
telegraf | disk,device=sda1,fstype=ext4,host=foobar,mode=rw,path=/etc/resolv.conf inodes_used=189723i,total=10499674112i,free=3154145280i,used=6791987200i,used_percent=68.2877210177679,inodes_total=655360i,inodes_free=465637i 1583406223000000000
telegraf | cpu,cpu=cpu-total,host=foobar usage_steal=0,usage_system=7.14285714285916,usage_softirq=0.8281573498966311,usage_nice=0,usage_iowait=0.10351966873709728,usage_irq=0,usage_guest=0,usage_guest_nice=0,usage_user=56.41821946171202,usage_idle=35.507246376802996 1583406223000000000
telegraf | cpu,cpu=cpu0,host=foobar usage_user=56.41821946171202,usage_irq=0,usage_steal=0,usage_system=7.14285714285916,usage_idle=35.507246376802996,usage_nice=0,usage_iowait=0.10351966873709728,usage_softirq=0.8281573498966311,usage_guest=0,usage_guest_nice=0 1583406223000000000
telegraf | processes,host=foobar sleeping=2i,paging=0i,idle=0i,dead=0i,total_threads=11i,blocked=0i,zombies=0i,stopped=0i,running=0i,total=2i,unknown=0i 1583406223000000000
telegraf | 2020-03-05T11:03:53Z D! [outputs.file] Wrote batch of 26 metrics in 1.676587ms
telegraf | 2020-03-05T11:03:53Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf | 2020-03-05T11:03:53Z D! [outputs.influxdb] Wrote batch of 26 metrics in 8.885827ms
telegraf | 2020-03-05T11:03:53Z D! [outputs.influxdb] Buffer fullness: 23 / 10000 metrics
在 Influxdb 中创建了我的传感器数据库
compose$ sudo docker exec -it influxdb influx
[sudo] password for jackbridger:
Connected to http://localhost:8086 version 1.7.10
InfluxDB shell version: 1.7.10
> show databases
name: databases
name
----
sensors
_internal
> use _internal
> show stats
name: runtime
Alloc Frees HeapAlloc HeapIdle HeapInUse HeapObjects HeapReleased HeapSys Lookups Mallocs NumGC NumGoroutine PauseTotalNs Sys TotalAlloc
----- ----- --------- -------- --------- ----------- ------------ ------- ------- ------- ----- ------------ ------------ --- ----------
16426120 26535115 16426120 44695552 21823488 225331 28131328 66519040 0 26760446 277 26 6132353 72483064 1337594448
name: queryExecutor
queriesActive queriesExecuted queriesFinished queryDurationNs recoveredPanics
------------- --------------- --------------- --------------- ---------------
1 3 2 3462237 0
name: database
tags: database=_internal
numMeasurements numSeries
--------------- ---------
12 467
...continues for a while.. can add more detail if it's helpfuL!
我尝试为 mosquitto 设置音量但无法正常工作 - 我认为这可能是个问题
任何人都可以提供任何指导，说明我如何调试它以弄清楚为什么 Telegraf 没有将添加到 MQTT 的消息发送到 Influxdb 中的传感器数据库吗？
谢谢！
朋友提供的以下内容非常有帮助，它解决了问题
# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
servers = ["tcp://localhost:1883"]
topics = [
"sensors"
]
data_format = "influx"
# ## MQTT broker URLs to be used. The format should be scheme://host:port,
# ## schema can be tcp, ssl, or ws.
# servers = ["tcp://127.0.0.1:1883"]
localhost:1883 会将主机解析为电报容器。如果你想在同一个 docker 组合设置中引用另一个容器 运行，你可以使用服务名称作为主机名，所以在这种情况下尝试 tcp://mosquitto:1883。在这里阅读：https://docs.docker.com/compose/networking/
每个容器现在可以查找主机名 web 或 db 并取回相应容器的 IP 地址。例如，Web 的应用程序代码可以连接到 URL postgres://db:5432 并开始使用 Postgres 数据库。
我正尝试按如下方式设置 TIG 堆栈：
传感器数据 -> MQTT (Eclipse-Mosquitto) -> Telegraf -> InfluxDB -> Grafana
我还没有设置传感器数据的来源，但在目前的状态下，一切都在编译并且 运行s 没有明显的错误，而且似乎都已连接但发布到 MQTT 主题的数据是没有出现在我的 Influxdb 数据库中。
$ sudo docker-compose up
Creating network "tig-samson-feature-setup-docker-compose_default" with the default driver
Creating mosquitto ... done
Creating influxdb ... done
Creating grafana ... done
Creating telegraf ... done
Attaching to influxdb, mosquitto, grafana, telegraf
mosquitto | 1583407265: mosquitto version 1.6.9 starting
mosquitto | 1583407265: Config loaded from /mosquitto/config/mosquitto.conf.
mosquitto | 1583407265: Opening ipv4 listen socket on port 1883.
mosquitto | 1583407265: Opening ipv6 listen socket on port 1883.
mosquitto | 1583407266: New connection from 192.168.32.1 on port 1883.
mosquitto | 1583407266: New client connected from 192.168.32.1 as mosqsub|5779-jackbridge (p1, c1, k60).
grafana | t=2020-03-05T11:21:05+0000 lvl=info msg="Starting Grafana" logger=server version=6.6.2 commit=3fa63cfc34 branch=HEAD compiled=2020-02-20T12:03:49+0000
grafana | t=2020-03-05T11:21:05+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini
grafana | t=2020-03-05T11:21:05+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini
grafana | t=2020-03-05T11:21:05+0000 lvl=info msg="Config overridden from command line" logger=settings arg="default.paths.data=/var/lib/grafana"
...continues indefinitely.. can add more detail if it's helpfuL!
下面是我的一些代码和我尝试过的一些东西
文件架构 运行在 ubuntu 虚拟机上（运行在 mac 上的 virtualbox 中）
docker-compose.yml
telegraf.conf
influxdb (folder)
-----data (folder)
-----------data (locked folder auto generated)
-----------meta (locked folder auto generated)
-----------wal (locked folder auto generated)
data
----mosquitto (empty folder)
docker-撰写
# # docker-compose.yml
version: '3'
services:
influxdb:
container_name: influxdb
image: influxdb
ports:
- "8086:8086"
environment:
- INFLUXDB_DB=sensors
- INFLUXDB_ADMIN_USER=admin
- INFLUXDB_ADMIN_PASSWORD=telegraf
volumes:
- ./influxdb/data:/var/lib/influxdb
restart: always
telegraf:
container_name: telegraf
image: telegraf
restart: always
depends_on:
- mosquitto
- influxdb
volumes:
- ./telegraf.conf:/etc/telegraf/telegraf.conf:ro
grafana:
container_name: grafana
image: grafana/grafana
links:
- influxdb
hostname: grafana
ports:
- "3000:3000"
mosquitto:
container_name: mosquitto
image: eclipse-mosquitto
ports:
- "1883:1883"
- "9001:9001"
restart: always
volumes:
- ./mosquitto.conf:/mqtt/config:ro
# - /data/mosquitto:mosquitto/data
（我用docker-compose up启动程序）
telegraf.conf
[[outputs.file]]
# ## Files to write to, "stdout" is a specially handled file.
files = ["stdout"]
# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
servers = ["tcp://localhost:1883"]
topics = [
"sensors"
]
data_format = "influx"
# ## MQTT broker URLs to be used. The format should be scheme://host:port,
# ## schema can be tcp, ssl, or ws.
# servers = ["tcp://127.0.0.1:1883"]
[[outputs.influxdb]]
urls = ["http://influxdb:8086"]
database = "sensors"
skip_database_creation = true
username = "telegraf"
password = "telegraf"
我可以访问 telegraf docker 容器并 ping influxdb 所以它似乎已连接
root@fc1d9af8f0c6:/# curl -sl -I http://influxdb:8086/ping
HTTP/1.1 204 No Content
Content-Type: application/json
Request-Id: 12b4923a-5ec7-11ea-8006-0242c0a81002
X-Influxdb-Build: OSS
X-Influxdb-Version: 1.7.10
X-Request-Id: 12b4923a-5ec7-11ea-8006-0242c0a81002
Date: Thu, 05 Mar 2020 09:52:49 GMT
我可以在其他终端发布订阅MQTT：
$ mosquitto_pub -t 'sensors' -m 'strength, site=room1 value=HELLO tony'
这是在订阅传感器的另一个终端中接收到的
$ mosquitto_sub -t 'sensors' -v # note this is executed before mosquitto_pub
sensors strength, site=room1 value=HELLO tony
当我执行 运行 mosquitto_sub 时，终端 运行ning docker-compose.activity 中没有 activity。
当我 运行 mosquitto_pub 客户端被检测为正在连接但发布事件似乎没有注册
mosquitto | 1583407850: New connection from 192.168.32.1 on port 1883.
mosquitto | 1583407850: New client connected from 192.168.32.1 as mosqpub|7039-jackbridge (p1, c1, k60).
mosquitto | 1583407850: Client mosqpub|7039-jackbridge disconnected.
Telegraf 似乎正在向 Influxdb 发送指标。在 telegraf [[outputs.file]] 中，我正在发送上面的文件 =['stdout']。目前尚不清楚指标是否到达 influxdb，因为我无法找到它们。
终端输出示例
telegraf | disk,device=sda1,fstype=ext4,host=foobar,mode=rw,path=/etc/resolv.conf inodes_used=189723i,total=10499674112i,free=3154145280i,used=6791987200i,used_percent=68.2877210177679,inodes_total=655360i,inodes_free=465637i 1583406223000000000
telegraf | cpu,cpu=cpu-total,host=foobar usage_steal=0,usage_system=7.14285714285916,usage_softirq=0.8281573498966311,usage_nice=0,usage_iowait=0.10351966873709728,usage_irq=0,usage_guest=0,usage_guest_nice=0,usage_user=56.41821946171202,usage_idle=35.507246376802996 1583406223000000000
telegraf | cpu,cpu=cpu0,host=foobar usage_user=56.41821946171202,usage_irq=0,usage_steal=0,usage_system=7.14285714285916,usage_idle=35.507246376802996,usage_nice=0,usage_iowait=0.10351966873709728,usage_softirq=0.8281573498966311,usage_guest=0,usage_guest_nice=0 1583406223000000000
telegraf | processes,host=foobar sleeping=2i,paging=0i,idle=0i,dead=0i,total_threads=11i,blocked=0i,zombies=0i,stopped=0i,running=0i,total=2i,unknown=0i 1583406223000000000
telegraf | 2020-03-05T11:03:53Z D! [outputs.file] Wrote batch of 26 metrics in 1.676587ms
telegraf | 2020-03-05T11:03:53Z D! [outputs.file] Buffer fullness: 0 / 10000 metrics
telegraf | 2020-03-05T11:03:53Z D! [outputs.influxdb] Wrote batch of 26 metrics in 8.885827ms
telegraf | 2020-03-05T11:03:53Z D! [outputs.influxdb] Buffer fullness: 23 / 10000 metrics
在 Influxdb 中创建了我的传感器数据库
compose$ sudo docker exec -it influxdb influx
[sudo] password for jackbridger:
Connected to http://localhost:8086 version 1.7.10
InfluxDB shell version: 1.7.10
> show databases
name: databases
name
----
sensors
_internal
> use _internal
> show stats
name: runtime
Alloc Frees HeapAlloc HeapIdle HeapInUse HeapObjects HeapReleased HeapSys Lookups Mallocs NumGC NumGoroutine PauseTotalNs Sys TotalAlloc
----- ----- --------- -------- --------- ----------- ------------ ------- ------- ------- ----- ------------ ------------ --- ----------
16426120 26535115 16426120 44695552 21823488 225331 28131328 66519040 0 26760446 277 26 6132353 72483064 1337594448
name: queryExecutor
queriesActive queriesExecuted queriesFinished queryDurationNs recoveredPanics
------------- --------------- --------------- --------------- ---------------
1 3 2 3462237 0
name: database
tags: database=_internal
numMeasurements numSeries
--------------- ---------
12 467
...continues for a while.. can add more detail if it's helpfuL!
我尝试为 mosquitto 设置音量但无法正常工作 - 我认为这可能是个问题
任何人都可以提供任何指导，说明我如何调试它以弄清楚为什么 Telegraf 没有将添加到 MQTT 的消息发送到 Influxdb 中的传感器数据库吗？
谢谢！
朋友提供的以下内容非常有帮助，它解决了问题
# # Read metrics from MQTT topic(s)
[[inputs.mqtt_consumer]]
servers = ["tcp://localhost:1883"]
topics = [
"sensors"
]
data_format = "influx"
# ## MQTT broker URLs to be used. The format should be scheme://host:port,
# ## schema can be tcp, ssl, or ws.
# servers = ["tcp://127.0.0.1:1883"]
localhost:1883 会将主机解析为电报容器。如果你想在同一个 docker 组合设置中引用另一个容器 运行，你可以使用服务名称作为主机名，所以在这种情况下尝试 tcp://mosquitto:1883。在这里阅读：https://docs.docker.com/compose/networking/
每个容器现在可以查找主机名 web 或 db 并取回相应容器的 IP 地址。例如，Web 的应用程序代码可以连接到 URL postgres://db:5432 并开始使用 Postgres 数据库。