Spark on Kubernetes with Minio - Postgres -> Minio - 由于以下原因无法创建执行程序

Question

您好，我在为 kubernetes 中的 spark-submit 提供依赖 jar 时遇到错误。

/usr/middleware/spark-3.1.1-bin-hadoop3.2/bin/spark-submit --master k8s://https://112.23.123.23:6443 --deploy-mode cluster --name spark-postgres-minio-kubernetes  --jars file:///AirflowData/kubernetes/externalJars/postgresql-42.2.14.jar  --driver-class-path file:///AirflowData/kubernetes/externalJars/postgresql-42.2.14.jar --conf spark.executor.instances=1 --conf spark.kubernetes.namespace=spark --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark --conf spark.kubernetes.file.upload.path=s3a://daci-dataintegration/spark-operator-on-k8s/code --conf spark.hadoop.fs.s3a.fast.upload=true --conf spark.kubernetes.container.image=hostname:5000/spark-py:spark3.1.2  file:///AirflowData/kubernetes/python/postgresminioKube.py

下面是要执行的代码。 S3 minio 和配置所需的 jar 放在 spark_home/conf 和 spark_home/jars 中，并创建 docker 图像。

from pyspark.sql import SparkSession
from pyspark.sql.functions import *
from pyspark.sql import functions as F
spark = SparkSession.builder.appName("Postgres-Minio-Kubernetes").getOrCreate()
import json
#spark = SparkSession.builder.config('spark.driver.extraClassPath', '/hadoop/externalJars/db2jcc4.jar').getOrCreate()
jdbcUrl = "jdbc:postgresql://{0}:{1}/{2}".format("hosnamme", "port", "db")
connectionProperties = {
  "user" : "username",
  "password" : "password",
  "driver": "org.postgresql.Driver",
  "fetchsize" : "100000"
}
pushdown_query = "(select * from public.employees) emp_als"
df = spark.read.jdbc(url=jdbcUrl, table=pushdown_query, column="employee_id", lowerBound=1, upperBound=100, numPartitions=2, properties=connectionProperties)
df.write.format('csv').options(delimiter=',').mode('overwrite').save('s3a://daci-dataintegration/spark-operator-on-k8s/data/postgres-minio-csv/')
df.write.format('parquet').options(delimiter='|').options(header=True).mode('overwrite').save('s3a://daci-dataintegration/spark-operator-on-k8s/data/postgres-minio-csv/')

错误如下。由于某种原因它正在尝试执行 jar

21/11/09 17:05:44 INFO SparkContext: Added JAR file:/tmp/spark-d987d7e7-9d49-4523-8415-1e438da1730e/postgresql-42.2.14.jar at spark://spark-postgres-minio-kubernetes-49d7d77d05a980e5-driver-svc.spark.svc:7078/jars/postgresql-42.2.14.jar with timestamp 1636477543573

21/11/09 17:05:49 ERROR TaskSchedulerImpl: Lost executor 1 on 192.168.216.12: Unable to create executor due to ./postgresql-42.2.14.jar

Answer 1

正在将外部 jar 添加到 /opt/spark/work-dir，但它没有访问权限。所以我更改了 dockerfile 以访问该文件夹然后它工作了。

RUN chmod 777 /opt/spark/work-dir

Spark on Kubernetes with Minio - Postgres -> Minio - 由于以下原因无法创建执行程序

Spark on Kubernetes with Minio - Postgres -> Minio -unable to create executor due to

apache-spark

kubernetes

pyspark

minio