使用 pyspark 从 redis 读取特定键

Question

我正在尝试使用 pyspark 从 Redis 中读取特定键。根据文档，我还没有找到任何特定的命令来读取特定的密钥。使用下面的代码我可以从 Redis 读取所有数据：

testid = spark.read.format("org.apache.spark.sql.redis")\
.option("table",'testing123')\
.option("key.column","id")\
.load()

求推荐

Answer 1

你可以试试keys.pattern。来自 docs:

To read Redis Hashes you have to provide a keys pattern with .option("keys.pattern", keysPattern) option. The DataFrame schema should be explicitly specified or can be inferred from a random row.

[...] Spark-Redis tries to extract the key based on the key pattern:

if the pattern ends with * and it's the only wildcard, the trailing substring will be extracted

otherwise there is no extraction - the key is kept as is.

testid = spark.read.format("org.apache.spark.sql.redis") \
.option("keys.pattern", "keyPattern:*") \
.option("key.column","id") \
.option("infer.schema", "true") \
.load()

使用 pyspark 从 redis 读取特定键

Read specific key from redis using pyspark

apache-spark

pyspark

redis

spark-redis