无法在运行 `sbt IntegrationTest/test` 和 HBaseTestingUtility.startMiniDFSCluster 时创建目录

Question

在 HBaseTestingUtility.startMiniDFSCluster 的帮助下在集成测试中创建 mini HDFS 集群 时，运行在 IntelliJ IDEA 中测试正常 但当运行通过 sbt IntegrationTest/test 时失败。错误如下所示：

22:00:38.430 [pool-5-thread-4] WARN  o.a.h.hdfs.server.namenode.NameNode - Encountered exception during format:
java.io.IOException: Cannot create directory /Users/jay/foobar/target/test-data/afd8c5d6-29a7-2a60-685a-d1c80c73a9c8/cluster_aa70cf12-8c75-2fd1-5602-e49c7026f79e/dfs/name-0-1/current
    at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:361)
    at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:571)
    at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:592)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:185)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1211)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:406)
    at org.apache.hadoop.hdfs.DFSTestUtil.formatNameNode(DFSTestUtil.java:233)
    at org.apache.hadoop.hdfs.MiniDFSCluster.configureNameService(MiniDFSCluster.java:1071)
    at org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:987)
    at org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:884)
    at org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:798)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:667)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniDFSCluster(HBaseTestingUtility.java:640)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1129)
    at org.apache.hadoop.hbase.HBaseTestingUtility.startMiniCluster(HBaseTestingUtility.java:1104)
    ...
...
22:00:38.435 [pool-5-thread-4] ERROR o.apache.hadoop.hdfs.MiniDFSCluster - IOE creating namenodes. Permissions dump:
path '/Users/jay/foobar/target/test-data/afd8c5d6-29a7-2a60-685a-d1c80c73a9c8/cluster_aa70cf12-8c75-2fd1-5602-e49c7026f79e/dfs/data':
    absolute:/Users/jay/foobar/target/test-data/afd8c5d6-29a7-2a60-685a-d1c80c73a9c8/cluster_aa70cf12-8c75-2fd1-5602-e49c7026f79e/dfs/data
    permissions: ----
    ...
...
[info] FooIntegrationTest:
[info] bar.foo.FooIntegrationTest *** ABORTED ***
[info]   java.io.IOException: Cannot create directory /Users/jay/foobar/target/test-data/afd8c5d6-29a7-2a60-685a-d1c80c73a9c8/cluster_aa70cf12-8c75-2fd1-5602-e49c7026f79e/dfs/name-0-1/current
...

Answer 1

测试套件的并行执行导致了问题。

在 build.sbt:

中将执行设置为串行

    IntegrationTest / parallelExecution := false, // Embedded HBase is having troubles when parallelled

无法在运行 `sbt IntegrationTest/test` 和 HBaseTestingUtility.startMiniDFSCluster 时创建目录

Cannot create directory while running `sbt IntegrationTest/test` with HBaseTestingUtility.startMiniDFSCluster

unit-testing

frameworks

sbt

hdfs

hbase

无法在 运行 `sbt IntegrationTest/test` 和 HBaseTestingUtility.startMiniDFSCluster 时创建目录

Cannot create directory while running `sbt IntegrationTest/test` with HBaseTestingUtility.startMiniDFSCluster

unit-testing

frameworks

sbt

hdfs

hbase

无法在运行 `sbt IntegrationTest/test` 和 HBaseTestingUtility.startMiniDFSCluster 时创建目录