什么可能导致 XUnit 测试批量失败但单独成功？

Question

我在VB.NET

中编写了以下括号平衡代码

Imports System.Collections.Generic

Public Module MatchingBrackets
    Private ReadOnly stack = New Stack(Of Char)
    Private ReadOnly closersOpeners = New Dictionary(Of Char, Char) From {
        {")"c, "("c},
        {"]"c, "["c},
        {"}"c, "{"c}
    }

    Public Function IsPaired(ByVal input As String) As Boolean
        For Each ch In input

            If closersOpeners.ContainsValue(ch) Then
                stack.Push(ch)
            ElseIf closersOpeners.ContainsKey(ch) Then
                If stack.Count > 0 AndAlso stack.Peek() = closersOpeners(ch) Then
                    stack.Pop()
                Else
                    Return False
                End If
            End If
        Next
        Dim result = stack.Count = 0
        stack.Clear()
        Return result
    End Function
End Module

我正在使用以下测试来测试这些

' This file was auto-generated based on version 2.0.0 of the canonical data.
Imports Xunit
Public Class MatchingBracketsTests
    <Fact>
    Public Sub PairedSquareBrackets()
        Dim value = "[]"
        Assert.True(IsPaired(value))
    End Sub
    <Fact>
    Public Sub EmptyString()
        Dim value = ""
        Assert.True(IsPaired(value))
    End Sub
    <Fact>
    Public Sub UnpairedBrackets()
        Dim value = "[["
        Assert.False(IsPaired(value))
    End Sub
    <Fact>
    Public Sub WrongOrderedBrackets()
        Dim value = "}{"
        Assert.False(IsPaired(value))
    End Sub
    <Fact>
    Public Sub WrongClosingBracket()
        Dim value = "{]"
        Assert.False(IsPaired(value))
    End Sub
    <Fact>
    Public Sub PairedWithWhitespace()
        Dim value = "{ }"
        Assert.True(IsPaired(value))
    End Sub
    <Fact>
    Public Sub PartiallyPairedBrackets()
        Dim value = "{[])"
        Assert.False(IsPaired(value))
    End Sub
    <Fact>
    Public Sub SimpleNestedBrackets()
        Dim value = "{[]}"
        Assert.True(IsPaired(value))
    End Sub
    <Fact>
    Public Sub SeveralPairedBrackets()
        Dim value = "{}[]"
        Assert.True(IsPaired(value))
    End Sub
    <Fact>
    Public Sub PairedAndNestedBrackets()
        Dim value = "([{}({}[])])"
        Assert.True(IsPaired(value))
    End Sub
    <Fact>
    Public Sub UnopenedClosingBrackets()
        Dim value = "{[)][]}"
        Assert.False(IsPaired(value))
    End Sub
    <Fact>
    Public Sub UnpairedAndNestedBrackets()
        Dim value = "([{])"
        Assert.False(IsPaired(value))
    End Sub
    <Fact>
    Public Sub PairedAndWrongNestedBrackets()
        Dim value = "[({]})"
        Assert.False(IsPaired(value))
    End Sub
    <Fact>
    Public Sub PairedAndIncompleteBrackets()
        Dim value = "{}["
        Assert.False(IsPaired(value))
    End Sub
    <Fact>
    Public Sub TooManyClosingBrackets()
        Dim value = "[]]"
        Assert.False(IsPaired(value))
    End Sub
    <Fact>
    Public Sub MathExpression()
        Dim value = "(((185 + 223.85) * 15) - 543)/2"
        Assert.True(IsPaired(value))
    End Sub
    <Fact>
    Public Sub ComplexLatexExpression()
        Dim value = "\left(\begin{array}{cc} \frac{1}{3} & x\ \mathrm{e}^{x} &... x^2 \end{array}\right)"
        Assert.True(IsPaired(value))
    End Sub
End Class

当我运行他们集体在Visual Studio或通过dotnet test时，三个失败，即

  Determining projects to restore...
  All projects are up-to-date for restore.
  MatchingBrackets -> D:\my_csharp\matching-brackets\bin\Debug\net5.0\MatchingBrackets.dll
Test run for D:\my_csharp\matching-brackets\bin\Debug\net5.0\MatchingBrackets.dll (.NETCoreApp,Version=v5.0)
Microsoft (R) Test Execution Command Line Tool Version 16.10.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
A total of 1 test files matched the specified pattern.
  Failed MatchingBrackets.MatchingBracketsTests.SimpleNestedBrackets [2 ms]
  Error Message:
   Assert.True() Failure
Expected: True
Actual:   False
  Stack Trace:
     at MatchingBrackets.MatchingBracketsTests.SimpleNestedBrackets() in D:\my_csharp\matching-brackets\MatchingBracketsTests.vb:line 42
  Failed MatchingBrackets.MatchingBracketsTests.PairedAndNestedBrackets [< 1 ms]
  Error Message:
   Assert.True() Failure
Expected: True
Actual:   False
  Stack Trace:
     at MatchingBrackets.MatchingBracketsTests.PairedAndNestedBrackets() in D:\my_csharp\matching-brackets\MatchingBracketsTests.vb:line 52
  Failed MatchingBrackets.MatchingBracketsTests.PairedWithWhitespace [< 1 ms]
  Error Message:
   Assert.True() Failure
Expected: True
Actual:   False
  Stack Trace:
     at MatchingBrackets.MatchingBracketsTests.PairedWithWhitespace() in D:\my_csharp\matching-brackets\MatchingBracketsTests.vb:line 32

Failed!  - Failed:     3, Passed:    14, Skipped:     0, Total:    17, Duration: 81 ms - MatchingBrackets.dll (net5.0)

但是，如果我运行每个失败的测试都单独进行，它们就不会失败。

知道为什么吗？

Answer 1

如果测试或正在测试的内容共享一个公共资源，该资源被每个测试影响同时执行的另一个测试的测试用例所操纵，通常会发生这种情况。

在这种特殊情况下，Stack 是所有这些测试之间的共享资源。

每次调用该函数都会清除堆栈，这会影响尚未完成的对该函数的任何其他调用。

如果重构为将堆栈作为局部变量，我认为应该隔离函数。

Public Module MatchingBrackets        
    Private ReadOnly closersOpeners = New Dictionary(Of Char, Char) From {
        {")"c, "("c},
        {"]"c, "["c},
        {"}"c, "{"c}
    }

    Public Function IsPaired(ByVal input As String) As Boolean
        Dim stack = New Stack(Of Char)
        For Each ch In input    
            If closersOpeners.ContainsValue(ch) Then
                stack.Push(ch)
            ElseIf closersOpeners.ContainsKey(ch) Then
                If stack.Count > 0 AndAlso stack.Peek() = closersOpeners(ch) Then
                    stack.Pop()
                Else
                    Return False
                End If
            End If
        Next
        Dim result = stack.Count = 0
        stack.Clear()
        Return result
    End Function
End Module

什么可能导致 XUnit 测试批量失败但单独成功？

What might cause an XUnit test to fail in a batch but succeed individually?

.net

vb.net

xunit