fiware quantumleap 插入 cratedb 不工作(模式丢失)
fiware quantumleap insert into cratedb not working (schema missing)
目标
使用 qunatumleap 将数据移动到 crate_db 以便稍后使用 Grafana 显示。
我做了什么
- 按照 tutorial 设置 Docker 图片
- 设置opc-agent以向 orion 经纪人提供数据
- 设置 quantumleap 以在更改时将数据从代理移动到 crate_db
- 已检查 contextBroker 中是否存在订阅
预期行为
订阅新项目 quantumleap 将在 crate_db 中的 table 中创建一个条目来存储提供的值
实际发生了什么
quantumleap 没有在 Crate_db 中创建条目,而是抛出了“模式不存在”错误。
所提供的教程没有谈论自己设置这些模式,因此我假设 quantumleap 通常会设置它们。
现在我不知道为什么会失败,很可能是我这边的配置错误
补充信息
- contextBroker 中存在订阅:
curl -X GET \
'http://localhost:1026/v2/subscriptions/' \
-H 'fiware-service: openiot' \
-H 'fiware-servicepath: /'
[
{"id":"60360eae34f0ca493f0fc148",
"description":"plc_id",
"status":"active",
"subject":{"entities":[{"idPattern":"PLC1"}],
"condition":{"attrs":["main"]}},
"notification":{"timesSent":1748,
"lastNotification":"2021-02-24T08:59:45.000Z",
"attrs":["main"],
"onlyChangedAttrs":false,
"attrsFormat":"normalized",
"http":{"url":"http://quantumleap:8668/v2/notify"},
"metadata":["dateCreated","dateModified"],
"lastSuccess":"2021-02-24T08:59:45.000Z",
"lastSuccessCode":500},
"throttling":1}
]
- 猎户座日志:
orion_1 | INFO@09:07:55 logTracing.cpp[130]: Request received: POST /v1/updateContext, request payload (327 bytes): {"contextElements":[{"type":"plc","isPattern":"false","id":"PLC1","attributes":[{"name":"main","type":"Number","value":"12285","metadatas":[{"name":"SourceTimestamp","type":"ISO8601","value":"2021-02-24T09:07:55.033Z"},{"name":"ServerTimestamp","type":"ISO8601","value":"2021-02-24T09:07:55.033Z"}]}]}],"updateAction":"UPDATE"}, response code: 200
- 飞跃日志:
quantumleap_1 | time=2021-02-24 09:07:55.125 | level=ERROR | corr=c7df320c-767f-11eb-bbb3-0242ac1b0005; cbnotif=1 | from=172.27.0.5 | srv=openiot | subserv=/ | op=_insert_entity_rows | comp=translators.crate | msg=Failed to insert entities because of below error; translator will still try saving original JSON in "mtopeniot"."etplc".__original_ngsi_entity__ | payload=[{'id': 'PLC1', 'type': 'plc', 'main': {'type': 'Number', 'value': '12285', 'metadata': {'dateCreated': {'type': 'DateTime', 'value': '2021-02-24T08:28:59.917Z'}, 'dateModified': {'type': 'DateTime', 'value': '2021-02-24T09:07:55.115Z'}}}, 'time_index': '2021-02-24T09:07:55.115000+00:00'}] | thread=140262103055136 | process=67
- 来自 Qunatumleap 的回溯
quantumleap_1 | Traceback (most recent call last): quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 365, in _insert_entity_rows
quantumleap_1 | self.cursor.executemany(stmt, rows) quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 67, in executemany quantumleap_1 | self.execute(sql, bulk_parameters=seq_of_parameters)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 53, in execute quantumleap_1 | self._result = self.connection.client.sql(sql, parameters,
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 331, in sql quantumleap_1 | content = self._json_request('POST', self.path, data=data)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 458, in _json_request quantumleap_1 | _raise_for_status(response)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 187, in _raise_for_status
quantumleap_1 | raise ProgrammingError(error.get('message', ''),
quantumleap_1 | crate.client.exceptions.ProgrammingError: SQLActionException[SchemaUnknownException: Schema 'mtopeniot' unknown] quantumleap_1 | quantumleap_1 | During handling of the above exception, another exception occurred: quantumleap_1 |
quantumleap_1 | Traceback (most recent call last):
quantumleap_1 | File "/src/ngsi-timeseries-api/src/reporter/reporter.py", line 195, in notify quantumleap_1 | trans.insert(payload, fiware_s, fiware_sp)
quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 221, in insert
quantumleap_1 | res = self._insert_entities_of_type(et,
quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 354, in _insert_entities_of_type
quantumleap_1 | self._insert_entity_rows(table_name, col_names, entries, entities) quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 381, in _insert_entity_rows
quantumleap_1 | self._insert_original_entities_in_failed_batch(
quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 437, in _insert_original_entities_in_failed_batch
quantumleap_1 | self.cursor.executemany(stmt, rows)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 67, in executemany
quantumleap_1 | self.execute(sql, bulk_parameters=seq_of_parameters)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 53, in execute
quantumleap_1 | self._result = self.connection.client.sql(sql, parameters,
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 331, in sql
quantumleap_1 | content = self._json_request('POST', self.path, data=data)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 458, in _json_request
quantumleap_1 | _raise_for_status(response) quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 187, in _raise_for_status
quantumleap_1 | raise ProgrammingError(error.get('message', ''),
quantumleap_1 | crate.client.exceptions.ProgrammingError: SQLActionException[SchemaUnknownException: Schema 'mtopeniot' unknown]
- 运行 qunatumleap 一段时间后 cratedb 中的表:
screenshot of cratedb tables
- docker-编写文件
version: "3"
services:
iotage:
hostname: iotage
image: iotagent4fiware/iotagent-opcua
networks:
- hostnet
- iotnet
ports:
- "4001:4001"
- "4081:8080"
extra_hosts:
- "iotcarsrv:192.168.2.16"
# - "PLC1:192.168.2.57"
depends_on:
- iotmongo
- orion
volumes:
- ./certificates:/opt/iotagent-opcua/certificates
- ./AGECONF:/opt/iotagent-opcua/conf
command: /usr/bin/tail -f /var/log/lastlog
iotmongo:
hostname: iotmongo
image: mongo:3.4
volumes:
- iotmongo_data:/data/db
- iotmongo_conf:/data/configdb
crate-db:
image: crate
hostname: crate-db
ports:
- "4200:4200"
- "4300:4300"
command:
crate -Clicense.enterprise=false -Cauth.host_based.enabled=false -Ccluster.name=democluster
-Chttp.cors.enabled=true -Chttp.cors.allow-origin="*"
networks:
- hostnet
quantumleap:
hostname: quantumleap
image: smartsdk/quantumleap
ports:
- "8668:8668"
depends_on:
- crate-db
environment:
- CRATE_HOST=crate-db
networks:
- hostnet
grafana:
image: grafana/grafana
depends_on:
- crate-db
ports:
- "3003:3000"
networks:
- hostnet
################ OCB ################
orion:
hostname: orion
image: fiware/orion:latest
networks:
- hostnet
- ocbnet
ports:
- "1026:1026"
depends_on:
- orion_mongo
#command: -dbhost mongo
entrypoint: /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -statCounters -dbhost mongo -logForHumans -logLevel DEBUG -t 255
orion_mongo:
hostname: orion_mongo
image: mongo:3.4
networks:
ocbnet:
aliases:
- mongo
volumes:
- orion_mongo_data:/data/db
- orion_mongo_conf:/data/configdb
command: --nojournal
volumes:
iotmongo_data:
iotmongo_conf:
orion_mongo_data:
orion_mongo_conf:
networks:
hostnet:
iotnet:
ocbnet:
编辑
- 已添加docker撰写文件
将数据库更改为更新版本(例如 crate-db:3.1.2)后,数据很好地到达数据库
目标
使用 qunatumleap 将数据移动到 crate_db 以便稍后使用 Grafana 显示。
我做了什么
- 按照 tutorial 设置 Docker 图片
- 设置opc-agent以向 orion 经纪人提供数据
- 设置 quantumleap 以在更改时将数据从代理移动到 crate_db
- 已检查 contextBroker 中是否存在订阅
预期行为
订阅新项目 quantumleap 将在 crate_db 中的 table 中创建一个条目来存储提供的值
实际发生了什么
quantumleap 没有在 Crate_db 中创建条目,而是抛出了“模式不存在”错误。 所提供的教程没有谈论自己设置这些模式,因此我假设 quantumleap 通常会设置它们。 现在我不知道为什么会失败,很可能是我这边的配置错误
补充信息
- contextBroker 中存在订阅:
curl -X GET \ 'http://localhost:1026/v2/subscriptions/' \ -H 'fiware-service: openiot' \ -H 'fiware-servicepath: /'
[
{"id":"60360eae34f0ca493f0fc148",
"description":"plc_id",
"status":"active",
"subject":{"entities":[{"idPattern":"PLC1"}],
"condition":{"attrs":["main"]}},
"notification":{"timesSent":1748,
"lastNotification":"2021-02-24T08:59:45.000Z",
"attrs":["main"],
"onlyChangedAttrs":false,
"attrsFormat":"normalized",
"http":{"url":"http://quantumleap:8668/v2/notify"},
"metadata":["dateCreated","dateModified"],
"lastSuccess":"2021-02-24T08:59:45.000Z",
"lastSuccessCode":500},
"throttling":1}
]
- 猎户座日志:
orion_1 | INFO@09:07:55 logTracing.cpp[130]: Request received: POST /v1/updateContext, request payload (327 bytes): {"contextElements":[{"type":"plc","isPattern":"false","id":"PLC1","attributes":[{"name":"main","type":"Number","value":"12285","metadatas":[{"name":"SourceTimestamp","type":"ISO8601","value":"2021-02-24T09:07:55.033Z"},{"name":"ServerTimestamp","type":"ISO8601","value":"2021-02-24T09:07:55.033Z"}]}]}],"updateAction":"UPDATE"}, response code: 200
- 飞跃日志:
quantumleap_1 | time=2021-02-24 09:07:55.125 | level=ERROR | corr=c7df320c-767f-11eb-bbb3-0242ac1b0005; cbnotif=1 | from=172.27.0.5 | srv=openiot | subserv=/ | op=_insert_entity_rows | comp=translators.crate | msg=Failed to insert entities because of below error; translator will still try saving original JSON in "mtopeniot"."etplc".__original_ngsi_entity__ | payload=[{'id': 'PLC1', 'type': 'plc', 'main': {'type': 'Number', 'value': '12285', 'metadata': {'dateCreated': {'type': 'DateTime', 'value': '2021-02-24T08:28:59.917Z'}, 'dateModified': {'type': 'DateTime', 'value': '2021-02-24T09:07:55.115Z'}}}, 'time_index': '2021-02-24T09:07:55.115000+00:00'}] | thread=140262103055136 | process=67
- 来自 Qunatumleap 的回溯
quantumleap_1 | Traceback (most recent call last): quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 365, in _insert_entity_rows
quantumleap_1 | self.cursor.executemany(stmt, rows) quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 67, in executemany quantumleap_1 | self.execute(sql, bulk_parameters=seq_of_parameters)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 53, in execute quantumleap_1 | self._result = self.connection.client.sql(sql, parameters,
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 331, in sql quantumleap_1 | content = self._json_request('POST', self.path, data=data)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 458, in _json_request quantumleap_1 | _raise_for_status(response)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 187, in _raise_for_status
quantumleap_1 | raise ProgrammingError(error.get('message', ''),
quantumleap_1 | crate.client.exceptions.ProgrammingError: SQLActionException[SchemaUnknownException: Schema 'mtopeniot' unknown] quantumleap_1 | quantumleap_1 | During handling of the above exception, another exception occurred: quantumleap_1 |
quantumleap_1 | Traceback (most recent call last):
quantumleap_1 | File "/src/ngsi-timeseries-api/src/reporter/reporter.py", line 195, in notify quantumleap_1 | trans.insert(payload, fiware_s, fiware_sp)
quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 221, in insert
quantumleap_1 | res = self._insert_entities_of_type(et,
quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 354, in _insert_entities_of_type
quantumleap_1 | self._insert_entity_rows(table_name, col_names, entries, entities) quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 381, in _insert_entity_rows
quantumleap_1 | self._insert_original_entities_in_failed_batch(
quantumleap_1 | File "/src/ngsi-timeseries-api/src/translators/sql_translator.py", line 437, in _insert_original_entities_in_failed_batch
quantumleap_1 | self.cursor.executemany(stmt, rows)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 67, in executemany
quantumleap_1 | self.execute(sql, bulk_parameters=seq_of_parameters)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/cursor.py", line 53, in execute
quantumleap_1 | self._result = self.connection.client.sql(sql, parameters,
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 331, in sql
quantumleap_1 | content = self._json_request('POST', self.path, data=data)
quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 458, in _json_request
quantumleap_1 | _raise_for_status(response) quantumleap_1 | File "/usr/local/lib/python3.8/site-packages/crate/client/http.py", line 187, in _raise_for_status
quantumleap_1 | raise ProgrammingError(error.get('message', ''),
quantumleap_1 | crate.client.exceptions.ProgrammingError: SQLActionException[SchemaUnknownException: Schema 'mtopeniot' unknown]
- 运行 qunatumleap 一段时间后 cratedb 中的表:
screenshot of cratedb tables
- docker-编写文件
version: "3"
services:
iotage:
hostname: iotage
image: iotagent4fiware/iotagent-opcua
networks:
- hostnet
- iotnet
ports:
- "4001:4001"
- "4081:8080"
extra_hosts:
- "iotcarsrv:192.168.2.16"
# - "PLC1:192.168.2.57"
depends_on:
- iotmongo
- orion
volumes:
- ./certificates:/opt/iotagent-opcua/certificates
- ./AGECONF:/opt/iotagent-opcua/conf
command: /usr/bin/tail -f /var/log/lastlog
iotmongo:
hostname: iotmongo
image: mongo:3.4
volumes:
- iotmongo_data:/data/db
- iotmongo_conf:/data/configdb
crate-db:
image: crate
hostname: crate-db
ports:
- "4200:4200"
- "4300:4300"
command:
crate -Clicense.enterprise=false -Cauth.host_based.enabled=false -Ccluster.name=democluster
-Chttp.cors.enabled=true -Chttp.cors.allow-origin="*"
networks:
- hostnet
quantumleap:
hostname: quantumleap
image: smartsdk/quantumleap
ports:
- "8668:8668"
depends_on:
- crate-db
environment:
- CRATE_HOST=crate-db
networks:
- hostnet
grafana:
image: grafana/grafana
depends_on:
- crate-db
ports:
- "3003:3000"
networks:
- hostnet
################ OCB ################
orion:
hostname: orion
image: fiware/orion:latest
networks:
- hostnet
- ocbnet
ports:
- "1026:1026"
depends_on:
- orion_mongo
#command: -dbhost mongo
entrypoint: /usr/bin/contextBroker -fg -multiservice -ngsiv1Autocast -statCounters -dbhost mongo -logForHumans -logLevel DEBUG -t 255
orion_mongo:
hostname: orion_mongo
image: mongo:3.4
networks:
ocbnet:
aliases:
- mongo
volumes:
- orion_mongo_data:/data/db
- orion_mongo_conf:/data/configdb
command: --nojournal
volumes:
iotmongo_data:
iotmongo_conf:
orion_mongo_data:
orion_mongo_conf:
networks:
hostnet:
iotnet:
ocbnet:
编辑
- 已添加docker撰写文件
将数据库更改为更新版本(例如 crate-db:3.1.2)后,数据很好地到达数据库