ksql table 中的数据不是持久的
Data in ksql table not being persistent
我们在 ubuntu 上使用融合平台。我们有简单的 JSON 数据通过 cURL 请求发送到名为 "UE_Context" 的 kafka 主题上的 kafka-rest 服务器。
使用以下命令为此主题创建名为 "UE_CONTEXT_STREAM" 的 kafka 流：
CREATE STREAM UE_Context_Stream (ue_key VARCHAR, ecgi VARCHAR) WITH (KAFKA_TOPIC='UE_Context', VALUE_FORMAT='JSON');
使用以下命令为此主题创建名为 "UE_CONTEXT_TABLE" 的 kafka table：
CREATE TABLE UE_Context_Table ( registertime BIGINT, ue_key VARCHAR, ecgi VARCHAR) WITH (KAFKA_TOPIC='UE_Context', KEY='ue_key', VALUE_FORMAT='JSON');
我使用以下 cURL 命令在主题上抽取了两行数据：
curl -X POST -H "Accept: application/json" -H "Content-Type: application/vnd.kafka.json.v1+json" --data '{"records":[{"key": "0x1234", "value":{"ue_key": "0x1234", "ecgi" : "1234"}}]}' "http://localhost:8082/topics/UE_Context"
curl -X POST -H "Accept: application/json" -H "Content-Type: application/vnd.kafka.json.v1+json" --data '{"records":[{"key": "0x1234", "value":{"ue_key": "0x4321", "ecgi" : "4321"}}]}' "http://localhost:8082/topics/UE_Context"
我有一个 select 查询正在等待 table，如下所示：
当 JSON 数据被注入主题时，此查询显示 table 信息。然后我们停止将 JSON 数据泵入主题并结束 select 查询并结束 select 查询。如果稍后执行 select，则不会显示先前填充的 table 信息。没有办法持久化这些数据吗？
Kafka 连接器和使用数据库可能是一种选择。但是 kSQL 没有临时内存来存储 table 信息吗？
a select is performed at a later point of time, the previously populated table info is not displayed.
一个select语句默认为主题的最新偏移量
如果您想查看以前的数据，you need to set the consumer offset back to the beginning
SET 'auto.offset.reset'='earliest';
此外，如文档中所述（强调）
A SELECT statement by itself is a non-persistent continuous query. The result of a SELECT statement isn’t persisted in a Kafka topic and is only printed in the KSQL console. Don’t confuse persistent queries created by CREATE STREAM AS SELECT with the streaming query result from a SELECT statement.
如 ksql github 自述文件所述：
ksqlDB allows you to define materialized views over your streams and
tables. Materialized views are defined by what is known as a
"persistent query". These queries are known as persistent because they
maintain their incrementally updated results using a table.
现在，关于实体化视图的信息比
持久查询，所以只是 read on:
The benefit of a materialized view is that it evaluates a query on the changes only (the delta),
instead of evaluating the query on the entire table.
...
In ksqlDB, a table can be materialized into a view or not. If a table is created
directly on top of a Kafka topic, it's not materialized.
Non-materialized tables can't be queried, because they would be highly
inefficient. On the other hand, if a table is derived from another
collection, ksqlDB materializes its results, and you can make queries
against it.
我们在 ubuntu 上使用融合平台。我们有简单的 JSON 数据通过 cURL 请求发送到名为 "UE_Context" 的 kafka 主题上的 kafka-rest 服务器。
使用以下命令为此主题创建名为 "UE_CONTEXT_STREAM" 的 kafka 流：
CREATE STREAM UE_Context_Stream (ue_key VARCHAR, ecgi VARCHAR) WITH (KAFKA_TOPIC='UE_Context', VALUE_FORMAT='JSON');
使用以下命令为此主题创建名为 "UE_CONTEXT_TABLE" 的 kafka table：
CREATE TABLE UE_Context_Table ( registertime BIGINT, ue_key VARCHAR, ecgi VARCHAR) WITH (KAFKA_TOPIC='UE_Context', KEY='ue_key', VALUE_FORMAT='JSON');
我使用以下 cURL 命令在主题上抽取了两行数据：
curl -X POST -H "Accept: application/json" -H "Content-Type: application/vnd.kafka.json.v1+json" --data '{"records":[{"key": "0x1234", "value":{"ue_key": "0x1234", "ecgi" : "1234"}}]}' "http://localhost:8082/topics/UE_Context"
curl -X POST -H "Accept: application/json" -H "Content-Type: application/vnd.kafka.json.v1+json" --data '{"records":[{"key": "0x1234", "value":{"ue_key": "0x4321", "ecgi" : "4321"}}]}' "http://localhost:8082/topics/UE_Context"
我有一个 select 查询正在等待 table，如下所示：
当 JSON 数据被注入主题时，此查询显示 table 信息。然后我们停止将 JSON 数据泵入主题并结束 select 查询并结束 select 查询。如果稍后执行 select，则不会显示先前填充的 table 信息。没有办法持久化这些数据吗？ Kafka 连接器和使用数据库可能是一种选择。但是 kSQL 没有临时内存来存储 table 信息吗？
a select is performed at a later point of time, the previously populated table info is not displayed.
一个select语句默认为主题的最新偏移量
如果您想查看以前的数据，you need to set the consumer offset back to the beginning
SET 'auto.offset.reset'='earliest';
此外，如文档中所述（强调）
A SELECT statement by itself is a non-persistent continuous query. The result of a SELECT statement isn’t persisted in a Kafka topic and is only printed in the KSQL console. Don’t confuse persistent queries created by CREATE STREAM AS SELECT with the streaming query result from a SELECT statement.
如 ksql github 自述文件所述：
ksqlDB allows you to define materialized views over your streams and tables. Materialized views are defined by what is known as a "persistent query". These queries are known as persistent because they maintain their incrementally updated results using a table.
现在，关于实体化视图的信息比 持久查询，所以只是 read on:
The benefit of a materialized view is that it evaluates a query on the changes only (the delta), instead of evaluating the query on the entire table. ...
In ksqlDB, a table can be materialized into a view or not. If a table is created directly on top of a Kafka topic, it's not materialized. Non-materialized tables can't be queried, because they would be highly inefficient. On the other hand, if a table is derived from another collection, ksqlDB materializes its results, and you can make queries against it.