ctest 报告测试成功，退出状态为 1

Question

我有一个带有单元测试的项目，当我运行 ctest（macos 上的 cmake 版本 3.18.2）时，所有测试都报告成功。但是，如果我运行单独进行其中一项测试，它的退出状态为 1。据我所知，这不应该发生，那么是什么原因造成的，我该如何解决？

这个问题是一个粗心的小错误，但是为了其他人的利益运行进入这个问题，我提供了一个答案，剩下的问题可以跳过

不幸的是，我无法用更小的最小工作示例重现此行为。我会尽量提供尽可能多的相关信息，如果我遗漏了什么请告诉我。这是单元测试的代码：

#include "alignment_reader.h"

#define CATCH_CONFIG_MAIN
#define CATCH_CONFIG_COLOUR_NONE
#include "catch.h"

#include "string_conversions.h"

#include "exceptions.h"

namespace paste_alignments {

namespace test {

namespace {

SCENARIO("Test correctness of AlignmentReader::FromFile.",
         "[AlignmentReader][FromFile][correctness]") {

  GIVEN("The name of a valid input file.") {
    std::string input_file{"test_data/valid_alignment_file.tsv"};

    WHEN("Constructed with default number of fields and chunk size.") {
      CHECK_NOTHROW(AlignmentReader::FromFile(input_file));
      /*AlignmentReader reader{AlignmentReader::FromFile(input_file)};

      THEN("Field number and chunk size are at default values.") {
        CHECK(reader.NumFields() == 12);
        CHECK(reader.ChunkSize() == 128 * 1000 * 1000);
      }*/
    }
  }
}

} // namespace

} // namespace test

} // namespace paste_alignments

当我取消注释部分时，退出代码变为 2，ctest 仍然报告成功。

这是我运行 ctest（有和没有评论部分）时发生的情况：

$ ctest
Test project /Users/Jasper/cpp_projects/PasteAlignments/debug
    Start 1: alignment_reader_test
1/1 Test #1: alignment_reader_test ............   Passed    0.16 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.17 sec

如果我单独运行测试并检查退出状态（注释掉的东西），我会得到以下结果：

$ ./test/alignment_reader_test 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
alignment_reader_test is a Catch v2.12.2 host application.
Run with -? for options

-------------------------------------------------------------------------------
Scenario: Test correctness of AlignmentReader::FromFile.
      Given: The name of a valid input file.
       When: Constructed with default number of fields and chunk size.
-------------------------------------------------------------------------------
/Users/Jasper/cpp_projects/PasteAlignments/test/alignment_reader_test.cc:64
...............................................................................

/Users/Jasper/cpp_projects/PasteAlignments/test/alignment_reader_test.cc:65: FAILED:
  CHECK_NOTHROW( AlignmentReader::FromFile(input_file) )
due to unexpected exception with message:
  Unable to open file: 'test_data/valid_alignment_file.tsv'.

===============================================================================
test cases: 1 | 1 failed
assertions: 1 | 1 failed

$ echo $?
1

当我包含注释部分时唯一改变的是两个测试失败并且退出状态是 2 而不是 1。

这里有 --verbose 标志：

$ ctest --verbose
UpdateCTestConfiguration  from :/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
Parse Config file:/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
UpdateCTestConfiguration  from :/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
Parse Config file:/Users/Jasper/cpp_projects/PasteAlignments/debug/DartConfiguration.tcl
Test project /Users/Jasper/cpp_projects/PasteAlignments/debug
Constructing a list of tests
Done constructing a list of tests
Updating test list for fixtures
Added 0 tests to meet fixture requirements
Checking test dependency graph...
Checking test dependency graph end
test 1
    Start 1: alignment_reader_test

1: Test command: /Users/Jasper/cpp_projects/PasteAlignments/debug/test/alignment_reader_test
1: Test timeout computed to be: 1500
1: ===============================================================================
1: All tests passed (1 assertion in 1 test case)
1: 
1/1 Test #1: alignment_reader_test ............   Passed    0.36 sec

100% tests passed, 0 tests failed out of 1

Total Test time (real) =   0.36 sec

我没有更改 ctest 配置（按照建议 here）。 CMakeLists.txt 看起来像这样：

add_executable(alignment_reader_test
        "${PROJECT_SOURCE_DIR}/test/alignment_reader_test.cc"
        "${PROJECT_SOURCE_DIR}/src/alignment_reader.cc"
        "${PROJECT_SOURCE_DIR}/src/alignment_batch.cc"
        "${PROJECT_SOURCE_DIR}/src/scoring_system.cc"
        "${PROJECT_SOURCE_DIR}/src/alignment.cc"
        "${PROJECT_SOURCE_DIR}/src/helpers.cc")
target_include_directories(alignment_reader_test PUBLIC
        "${PROJECT_SOURCE_DIR}/test"
        "${PROJECT_SOURCE_DIR}/include"
        "${PROJECT_SOURCE_DIR}/lib/catch/include")
add_test(NAME alignment_reader_test COMMAND alignment_reader_test)

Answer 1

感谢@Tsyvarev，我发现了这个问题。

build目录结构如下：

build/
|--test/
   |--test_executable
   |--test_data/
      |--test_datafile

当 CMake 运行s test_executable 时，它运行它来自目录 test.

当我单独运行可执行文件时，我运行它来自 build 目录。 (./test/test_executable).

在 unit-test 代码中，test_datafile 被称为 test_data/test_datafile。当测试来自 build 可执行文件（就像我所做的那样）时运行而不是来自 build/test 目录的运行时无法识别。

因此，当ctest 运行测试时，它实际上应该成功了。

确实，如果我先 cd 进入 test，测试的退出代码应该是 0。

ctest 报告测试成功，退出状态为 1

ctest reports success for test with exit status 1

c++

cmake

ctest

exitstatus

catch2