使用 Spark 读取 S3 文件时出现 NullPointerException

Question

我正在尝试使用 Spark 读取 S3 文件并遇到以下异常：

java.lang.NullPointerException
    at org.apache.hadoop.fs.s3native.NativeS3FileSystem.getFileStatus(NativeS3FileSystem.java:433)
    at org.apache.hadoop.fs.Globber.getFileStatus(Globber.java:57)
    at org.apache.hadoop.fs.Globber.glob(Globber.java:248)
    at org.apache.hadoop.fs.FileSystem.globStatus(FileSystem.java:1642)
    at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:257)
    at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)
    at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:304)
    at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:201)
    at org.apache.spark.rdd.RDD$anonfun$partitions.apply(RDD.scala:205)
    at org.apache.spark.rdd.RDD$anonfun$partitions.apply(RDD.scala:203)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
    at org.apache.spark.rdd.MappedRDD.getPartitions(MappedRDD.scala:28)
    at org.apache.spark.rdd.RDD$anonfun$partitions.apply(RDD.scala:205)
    at org.apache.spark.rdd.RDD$anonfun$partitions.apply(RDD.scala:203)
    at scala.Option.getOrElse(Option.scala:120)
    at org.apache.spark.rdd.RDD.partitions(RDD.scala:203)
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1328)
    at org.apache.spark.rdd.RDD.count(RDD.scala:910)
    at $iwC$iwC$iwC$iwC.<init>(<console>:13)
    at $iwC$iwC$iwC.<init>(<console>:18)
    at $iwC$iwC.<init>(<console>:20)
    at $iwC.<init>(<console>:22)
    at <init>(<console>:24)
    at .<init>(<console>:28)
    at .<clinit>(<console>)
    at .<init>(<console>:7)
    at .<clinit>(<console>)
    at $print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:852)
    at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1125)
    at org.apache.spark.repl.SparkIMain.loadAndRunReq(SparkIMain.scala:674)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:705)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:669)
    at org.apache.spark.repl.SparkILoop.reallyInterpret(SparkILoop.scala:828)
    at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:873)
    at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:785)
    at org.apache.spark.repl.SparkILoop.processLine(SparkILoop.scala:628)
    at org.apache.spark.repl.SparkILoop.innerLoop(SparkILoop.scala:636)
    at org.apache.spark.repl.SparkILoop.loop(SparkILoop.scala:641)
    at org.apache.spark.repl.SparkILoop$anonfun$process.apply$mcZ$sp(SparkILoop.scala:968)
    at org.apache.spark.repl.SparkILoop$anonfun$process.apply(SparkILoop.scala:916)
    at org.apache.spark.repl.SparkILoop$anonfun$process.apply(SparkILoop.scala:916)
    at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:916)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1011)
    at org.apache.spark.repl.Main$.main(Main.scala:31)
    at org.apache.spark.repl.Main.main(Main.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:358)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:75)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

它只是一个文本文件，我正在尝试使用以下代码行 (spark-shell) 读取它：

sc.textFile("s3n://<bucket>/mypath/file.csv").count

凭据（fs.s3n.awsAccessKeyId 和 fs.s3n.awsSecretAccessKey）已使用临时核心-site.xml 文件正确设置（机器上未安装 Hadoop）。

我错过了什么？

Answer 1

这是权限问题。默认情况下，权限仅授予 AWS 用户。如果您使用带访问密钥的 IAM 身份验证，则必须在 S3 中向 "authenticated users" 添加权限。

使用 Spark 读取 S3 文件时出现 NullPointerException

Getting NullPointerException when reading an S3 file with Spark

hadoop

amazon-s3

apache-spark