如何遍历 Scala 列表中的每个字典？

Question

我有一个 JSON 字符串，它基本上是一个字典列表。我正在尝试使用 for 循环来遍历它。基本上在每次迭代中，我应该能够得到index、name、source等键, s3。有人可以帮忙吗？所以在第一次迭代中我应该能够得到 index=1,source=a 等

import scala.util.parsing.json._
class CC[T] { def unapply(a:Any):Option[T] = Some(a.asInstanceOf[T]) }

object M extends CC[Map[String, Any]]
object A extends CC[List[Any]] //for s3
object I extends CC[Double]
object S extends CC[String]
object E extends CC[String]
object F extends CC[String]
object G extends CC[Map[String, Any]]

val jsonString =
    """
    [{
        "index": 1,
      "source": "a",
      "name": "v",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     },
     {
        "index": 2,
      "source": "b",
      "name": "b",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     }]
     
    """.stripMargin
    
println(List(JSON.parseFull(jsonString)) )

Answer 1

如果你想不用spark，那么你可以这样试试：

JSON.parseFull(jsonString) match {
    case Some(x) => {
        val json_list = x.asInstanceOf[List[Map[String, Any]]]
        json_list.foreach { json_map =>
            // To extract any base level fields

            // You can do any action on the extracted field. This is just to illustrate the extraction method
            println("index: " + json_map("index"))
            println("source: " + json_map("source"))

            // To extract fields from inside a list: example element at index 0 of list s3

            // You can do any action on the extracted field. This is just to illustrate the extraction method
            val s3_list = json_map("s3").asInstanceOf[List[Any]]
            println("s3: " + s3_list(0))

            // To extract fields from inside a map: example path from 0th element of list s3

            // You can do any action on the extracted field. This is just to illustrate the extraction method
            println("s3.path: " + s3_list(0).asInstanceOf[Map[String, Any]]("path"))

        }
    }
    case None => {
        println("Improper JSON")
    }
}

// Output of the print statements

// Iteration #1:

index: 1.0
source: a
s3: Map(path -> s3://1, bucket -> p, key -> r)
s3.path: s3://1

// Iteration #2:

index: 2.0
source: b
s3: Map(path -> s3://1, bucket -> p, key -> r)
s3.path: s3://1

Answer 2

在您的示例中，您尝试使用提取器。基于 unapply 方法的提取器。通常它需要 objet 和 return it 参数。要实现这一点，您需要为要使用的每种类型定义提取器，但不需要像您的示例 S,E,F 那样复制它们。将对象提取为特定类型后，您可以对其进行操作。

  import scala.collection.Map
  import scala.util.parsing.json._

  class CC[T] {
    def unapply(a: Any): Option[T] = Some(a.asInstanceOf[T])
  }

  object M extends CC[Map[String, Any]]
  object L extends CC[List[Any]]
  object S extends CC[String]
  object D extends CC[Double]

  val jsonString2 =
    """
    [{
        "index": 1,
      "source": "a",
      "name": "v",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     },
     {
        "index": 2,
      "source": "b",
      "name": "b",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     }]

    """.stripMargin

  val result = for {
    Some(L(list)) <- List(JSON.parseFull(jsonString2))
    M(map) <- list
    D(index) = map("index")
    S(source) = map("source")
    S(name) = map("name")
    L(s3Connections) = map("s3")
    M(s3) <- s3Connections
    S(path) = s3("path")
    S(bucket) = s3("bucket")
    S(key) = s3("key")
  } yield (index, source, name, path, bucket, key)

  println(result)
  
//  List((1.0,a,v,s3://1,p,r), (2.0,b,b,s3://1,p,r))

此外，您可以使用任何库来解析 json 文件。例如，我在下面提供了使用 circe 库的示例。

  import io.circe.parser.parse
  import io.circe.{Decoder, Json}
  
  case class S3Config(path: String, bucket: String, key: String)

  object S3Config {
    implicit val decoder: Decoder[S3Config] = deriveDecoder[S3Config]
  }

  case class Source(index: Int, source: String, name: String, s3: Vector[S3Config])

  object Source {
    implicit val decoder: Decoder[Source] = deriveDecoder[Source]
  }

  val jsonString =
    """
    [{
        "index": 1,
      "source": "a",
      "name": "v",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     },
     {
        "index": 2,
      "source": "b",
      "name": "b",
      "s3": [{
        "path": "s3://1",
        "bucket": "p",
        "key": "r"
      }]
     }]

    """.stripMargin

  val sources = parse(jsonString).getOrElse(Json.Null).as[Vector[Source]].getOrElse(Vector.empty[Source])
  for (source <- sources) {
    println(source.source)
    println(source.index)
    println(source.s3)
  }

  //  a
  //  1
  //  Vector(S3Config(s3://1,p,r))
  //      b
  //  2
  //  Vector(S3Config(s3://1,p,r))

如何遍历 Scala 列表中的每个字典？

How to iterate though each dictionary from the list in Scala?

scala

apache-spark

pyspark

apache-spark-sql