我可以在 dataframe 上应用 AWS FindMatch 转换吗？如果是那么如何

Question

我想了解是否可以在 AWS Glue 中对 spark 数据帧应用 FindMatch ml 转换。目前我可以在动态框架上使用它。如果我想在动态帧上使用 findmatch 转换，下面是语法。

<output DynamicFrame on which the ml transform has been applied> = 
FindMatches.apply(frame = <Input DynamicFrame>, transformId = <transformation 
id of the findmatch ml transform created separately>)

我尝试使用数据帧代替输入动态帧，当我 运行 Glue 作业时，它失败了。错误显示如下

"Attribute Error: 'DataFrame' object has no attribute 'glue_ctx'"

下面是我尝试使用数据框的代码

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglueml.transforms import FindMatches

args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "hospitality", table_name = 
"personinputdata", transformation_ctx = "datasource0")
df0 = datasource0.toDF()

resolvechoice1 = ResolveChoice.apply(frame = datasource0, choice = "MATCH_CATALOG", database = 
"hospitality", table_name = "personinputdata", transformation_ctx = "resolvechoice1")

findmatchdf = FindMatches.apply(frame = df0, transformId = "tfm- 
01cc9b02c93640cfc7ce5ea91745e24258cb2e01")
findmatchdf.show()

下面是我尝试使用动态帧而不是数据帧时的代码，并且代码有效。

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglueml.transforms import FindMatches

args = getResolvedOptions(sys.argv, ['JOB_NAME'])

sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

datasource0 = glueContext.create_dynamic_frame.from_catalog(database = "hospitality", table_name = 
"patientinputdata", transformation_ctx = "datasource0")

resolvechoice1 = ResolveChoice.apply(frame = datasource0, choice = "MATCH_CATALOG", database = 
"hospitality", table_name = "patientinputdata", transformation_ctx = "resolvechoice1")

findmatches2 = FindMatches.apply(frame = resolvechoice1, transformId = "tfm- 
0cadd1e6d2da40d7c18db7836e92be93833b6019", transformation_ctx = "findmatches2")

我尝试在线搜索是否可以找到 FindMatch ml 转换的代码，但在任何地方都找不到。

Answer 1

如您所知，FindMatch 仅适用于动态帧... 因此，您可以随时将 spark df 转换为动态框架 运行 it

from awsglue.dynamicframe import DynamicFrame
glueContext = GlueContext(SparkContext.getOrCreate())
Dyf0 = DynamicFrame.fromDF(df0, glueContext, "anyname")

然后 运行 根据需要进行 FindMatch。