UI 在 CI 上测试 Flaky
UI Tests Flaky on CI
更新：
VNC 进入构建机器（不做任何更改）以某种方式修复了这个....奇怪
上下文
我们使用 Mac EC2 实例 CI 集群进行 运行 UI 测试。集群设置为每晚回收。意味着新集群在早上配置。我们注意到测试在一天中变得越来越不稳定。
- Build 312 是早上的第一个 build。
- 红色是失败的测试，橙色是静音测试，黄色是片状测试，灰色是未知状态
- 提交只是白色 space 触发构建的自动提交。测试内容相同
- 注意：Bluepill 配置为 运行 每台机器一个模拟器（无并发）
日志
不稳定的测试或失败通常是由于“等待某些资源空闲”并达到超时，或某些网络请求完成。失败或不稳定的测试也有一个共同点，即它们通常与视频有关
这里有几个示例日志
正在等待应用空闲
Timed out waiting for app to idle.
The following idling resources are busy.
1. GREYAppStateTracker:
Waiting for network requests to finish. By default, EarlGrey tracks all network requests. To change this behavior, refer to GREYConfiguration.
<__NSCFLocalDataTask:0x7fcfcfd87450, URL:"https://v.myorgimg.com/videos/mc/hls/e7/01/ca/e701caebac12684056eb699f316fc411_360w.m3u8"> => Waiting for network requests to finish. By default, EarlGrey tracks all network requests. To change this behavior, refer to GREYConfiguration.
(
0 MyorgDevelopmentEG2 0x000000010ebf3205 -[GREYAppStateTrackerObject setState:] + 69
1 MyorgDevelopmentEG2 0x000000010ebf22b3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
2 MyorgDevelopmentEG2 0x000000010ebf18f0 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
3 MyorgDevelopmentEG2 0x000000010ebf1e34 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:] + 1076
4 MyorgDevelopmentEG2 0x000000010ebf0ac6 -[GREYAppStateTracker trackState:forObject:] + 102
5 MyorgDevelopmentEG2 0x000000010ebca867 -[__NSCFLocalDataTask_GREYApp grey_track] + 135
6 MyorgDevelopmentEG2 0x000000010ebcac55 -[__NSCFLocalDataTask_GREYApp greyswizzled_resume] + 181
7 MediaToolbox 0x00007fff2b617285 figHttpRequestSetupNSURLSessionTask + 1380
8 MediaToolbox 0x00007fff2b616bb6 _FigHTTPRequestCreateWithNSURLSession + 2923
9 MediaToolbox 0x00007fff2b64aaaf figHTTPRequestSessionNSCreateHTTPRequest + 199
10 MediaToolbox 0x00007fff2b71f5b1 segPumpCreateHTTPRequest + 944
11 MediaToolbox 0x00007fff2b721de6 segPumpSendIndexFileRequest + 1703
12 MediaToolbox 0x00007fff2b7431d5 segPumpRequestIndexForStream + 753
13 MediaToolbox 0x00007fff2b7421ee segPumpSetAlternateForStream + 3145
14 MediaToolbox 0x00007fff2b73e9e5 segPumpSetCurrentAlternate + 9898
15 MediaToolbox 0x00007fff2b791d6a ProduceStreamingAssetProperty + 661
16 MediaToolbox 0x00007fff2b78e07c URLAssetPropertyWorkFunction + 872
17 libdispatch.dylib 0x00007fff201078df _dispatch_client_callout + 8
18 libdispatch.dylib 0x00007fff2010de15 _dispatch_lane_serial_drain + 715
19 libdispatch.dylib 0x00007fff2010e9c3 _dispatch_lane_invoke + 455
20 libdispatch.dylib 0x00007fff20117a8e _dispatch_root_queue_drain + 350
21 libdispatch.dylib 0x00007fff2011786c _dispatch_worker_thread + 222
22 libsystem_pthread.dylib 0x00007fff60342950 _pthread_start + 224
23 libsystem_pthread.dylib 0x00007fff6033e47b thread_start + 15
)
正在等待网络请求完成
9. GREYAppStateTracker:
Waiting for network requests to finish. By default, EarlGrey tracks all network requests. To change this behavior, refer to GREYConfiguration.
Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
<ASCollectionView: 0x7feca21ad000> => Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
(
0 MyorgDevelopmentEG2 0x000000010bb2e335 -[GREYAppStateTrackerObject setState:] + 69
1 MyorgDevelopmentEG2 0x000000010bb2d3e3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
2 MyorgDevelopmentEG2 0x000000010bb2ca20 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
3 MyorgDevelopmentEG2 0x000000010bb2cf64 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:] + 1076
4 MyorgDevelopmentEG2 0x000000010bb2bbf6 -[GREYAppStateTracker trackState:forObject:] + 102
5 MyorgDevelopmentEG2 0x000000010bb02b0d -[UIView(GREYApp) greyswizzled_setNeedsLayout] + 93
6 UIKitCore 0x00007fff24ba5675 -[UIScrollView setNeedsLayout] + 79
7 UIKitCore 0x00007fff23dc0390 -[UICollectionView _updateAnimationDidStop:finished:context:] + 2236
8 UIKitCore 0x00007fff23dbf9e4 __102-[UICollectionView _updateWithItems:tentativelyForReordering:propertyAnimator:collectionViewAnimator:]_block_invoke.2110 + 77
9 MyorgDevelopmentEG2 0x000000010bb2fdef __60-[GREYDispatchQueueTracker grey_dispatchAsyncCallWithBlock:]_block_invoke + 47
10 libdispatch.dylib 0x00007fff2010670d _dispatch_call_block_and_release + 12
11 libdispatch.dylib 0x00007fff201078df _dispatch_client_callout + 8
12 libdispatch.dylib 0x00007fff20114a27 _dispatch_main_queue_callback_4CF + 1045
13 CoreFoundation 0x00007fff203908f8 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 9
14 CoreFoundation 0x00007fff2038b169 __CFRunLoopRun + 2781
15 CoreFoundation 0x00007fff2038a1a7 CFRunLoopRunSpecific + 567
16 GraphicsServices 0x00007fff2b874d85 GSEventRunModal + 139
17 UIKitCore 0x00007fff246c14df -[UIApplication _run] + 912
18 UIKitCore 0x00007fff246c639c UIApplicationMain + 101
19 MyorgDevelopmentEG2 0x0000000108f175de main + 190
20 libdyld.dylib 0x00007fff2025abbd start + 1
21 ??? 0x0000000000000005 0x0 + 5
)
<_ASDisplayLayer: 0x600007996580> => Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
假设
这让我认为有一些孤立的子进程或守护进程仍然保留着资源。或者有工件没有被正确清理。
我尝试过的东西
- 关闭、擦除、删除构建之间的所有模拟器（无帮助）
xcrun simctl shutdown all
xcrun simctl erase all
xcrun simctl delete all
osascript -e 'tell application "iOS Simulator" to quit'
osascript -e 'tell application "Simulator" to quit'
- 在构建之间终止 CoreSimulator 相关服务（无帮助）
launchctl remove com.apple.CoreSimulator.CoreSimulatorService || true
launchctl remove com.apple.CoreSimulator.SimLaunchHost-x86 || true
launchctl remove com.apple.CoreSimulator.SimulatorTrampoline || true
launchctl remove com.apple.CoreSimulator.SimLaunchHost-arm64 || true
- 删除生成之间的模拟器设备、日志和缓存。这实际上有点帮助。薄片最初消失了，但在几次构建后很快又回来了，而机器仍在构建之间执行清洁....不确定为什么。
echo "Delete all simulator devices, cache, and temp"
rm -rf ~/Library/Developer/CoreSimulator
echo "Delete Simulator Logs"
rm -rf ~/Library/Logs/CoreSimulator
- 在 EarlGrey
上增加 Animation Timeout, Interaction Timeout
- Blacklist the URL 请求挂起。这确实通过忽略症状而不是修复实际问题来提高测试套件的稳定性，所以我希望我可以避免这种情况。
还有其他人 运行遇到同样的问题吗？感谢有人能对此有所了解或提出尝试建议。
注意：我们在 Buildkite CI 管道上使用 Bluepill and EarlGrey2 到 运行 UI 测试（iOS 14.5，Xcode 12.5）。
你试过了吗运行:
sudo systemsetup -setcomputersleep never
更新：
VNC 进入构建机器（不做任何更改）以某种方式修复了这个....奇怪
上下文
我们使用 Mac EC2 实例 CI 集群进行 运行 UI 测试。集群设置为每晚回收。意味着新集群在早上配置。我们注意到测试在一天中变得越来越不稳定。
- Build 312 是早上的第一个 build。
- 红色是失败的测试，橙色是静音测试，黄色是片状测试，灰色是未知状态
- 提交只是白色 space 触发构建的自动提交。测试内容相同
- 注意：Bluepill 配置为 运行 每台机器一个模拟器（无并发）
日志
不稳定的测试或失败通常是由于“等待某些资源空闲”并达到超时，或某些网络请求完成。失败或不稳定的测试也有一个共同点，即它们通常与视频有关
这里有几个示例日志
正在等待应用空闲
Timed out waiting for app to idle.
The following idling resources are busy.
1. GREYAppStateTracker:
Waiting for network requests to finish. By default, EarlGrey tracks all network requests. To change this behavior, refer to GREYConfiguration.
<__NSCFLocalDataTask:0x7fcfcfd87450, URL:"https://v.myorgimg.com/videos/mc/hls/e7/01/ca/e701caebac12684056eb699f316fc411_360w.m3u8"> => Waiting for network requests to finish. By default, EarlGrey tracks all network requests. To change this behavior, refer to GREYConfiguration.
(
0 MyorgDevelopmentEG2 0x000000010ebf3205 -[GREYAppStateTrackerObject setState:] + 69
1 MyorgDevelopmentEG2 0x000000010ebf22b3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
2 MyorgDevelopmentEG2 0x000000010ebf18f0 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
3 MyorgDevelopmentEG2 0x000000010ebf1e34 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:] + 1076
4 MyorgDevelopmentEG2 0x000000010ebf0ac6 -[GREYAppStateTracker trackState:forObject:] + 102
5 MyorgDevelopmentEG2 0x000000010ebca867 -[__NSCFLocalDataTask_GREYApp grey_track] + 135
6 MyorgDevelopmentEG2 0x000000010ebcac55 -[__NSCFLocalDataTask_GREYApp greyswizzled_resume] + 181
7 MediaToolbox 0x00007fff2b617285 figHttpRequestSetupNSURLSessionTask + 1380
8 MediaToolbox 0x00007fff2b616bb6 _FigHTTPRequestCreateWithNSURLSession + 2923
9 MediaToolbox 0x00007fff2b64aaaf figHTTPRequestSessionNSCreateHTTPRequest + 199
10 MediaToolbox 0x00007fff2b71f5b1 segPumpCreateHTTPRequest + 944
11 MediaToolbox 0x00007fff2b721de6 segPumpSendIndexFileRequest + 1703
12 MediaToolbox 0x00007fff2b7431d5 segPumpRequestIndexForStream + 753
13 MediaToolbox 0x00007fff2b7421ee segPumpSetAlternateForStream + 3145
14 MediaToolbox 0x00007fff2b73e9e5 segPumpSetCurrentAlternate + 9898
15 MediaToolbox 0x00007fff2b791d6a ProduceStreamingAssetProperty + 661
16 MediaToolbox 0x00007fff2b78e07c URLAssetPropertyWorkFunction + 872
17 libdispatch.dylib 0x00007fff201078df _dispatch_client_callout + 8
18 libdispatch.dylib 0x00007fff2010de15 _dispatch_lane_serial_drain + 715
19 libdispatch.dylib 0x00007fff2010e9c3 _dispatch_lane_invoke + 455
20 libdispatch.dylib 0x00007fff20117a8e _dispatch_root_queue_drain + 350
21 libdispatch.dylib 0x00007fff2011786c _dispatch_worker_thread + 222
22 libsystem_pthread.dylib 0x00007fff60342950 _pthread_start + 224
23 libsystem_pthread.dylib 0x00007fff6033e47b thread_start + 15
)
正在等待网络请求完成
9. GREYAppStateTracker:
Waiting for network requests to finish. By default, EarlGrey tracks all network requests. To change this behavior, refer to GREYConfiguration.
Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
<ASCollectionView: 0x7feca21ad000> => Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
(
0 MyorgDevelopmentEG2 0x000000010bb2e335 -[GREYAppStateTrackerObject setState:] + 69
1 MyorgDevelopmentEG2 0x000000010bb2d3e3 __133-[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:]_block_invoke + 931
2 MyorgDevelopmentEG2 0x000000010bb2ca20 -[GREYAppStateTracker grey_performBlockInCriticalSection:] + 224
3 MyorgDevelopmentEG2 0x000000010bb2cf64 -[GREYAppStateTracker grey_changeState:usingOperation:forObject:orInternalObjectDeallocationTracker:orExternalAppStateTrackerObject:] + 1076
4 MyorgDevelopmentEG2 0x000000010bb2bbf6 -[GREYAppStateTracker trackState:forObject:] + 102
5 MyorgDevelopmentEG2 0x000000010bb02b0d -[UIView(GREYApp) greyswizzled_setNeedsLayout] + 93
6 UIKitCore 0x00007fff24ba5675 -[UIScrollView setNeedsLayout] + 79
7 UIKitCore 0x00007fff23dc0390 -[UICollectionView _updateAnimationDidStop:finished:context:] + 2236
8 UIKitCore 0x00007fff23dbf9e4 __102-[UICollectionView _updateWithItems:tentativelyForReordering:propertyAnimator:collectionViewAnimator:]_block_invoke.2110 + 77
9 MyorgDevelopmentEG2 0x000000010bb2fdef __60-[GREYDispatchQueueTracker grey_dispatchAsyncCallWithBlock:]_block_invoke + 47
10 libdispatch.dylib 0x00007fff2010670d _dispatch_call_block_and_release + 12
11 libdispatch.dylib 0x00007fff201078df _dispatch_client_callout + 8
12 libdispatch.dylib 0x00007fff20114a27 _dispatch_main_queue_callback_4CF + 1045
13 CoreFoundation 0x00007fff203908f8 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 9
14 CoreFoundation 0x00007fff2038b169 __CFRunLoopRun + 2781
15 CoreFoundation 0x00007fff2038a1a7 CFRunLoopRunSpecific + 567
16 GraphicsServices 0x00007fff2b874d85 GSEventRunModal + 139
17 UIKitCore 0x00007fff246c14df -[UIApplication _run] + 912
18 UIKitCore 0x00007fff246c639c UIApplicationMain + 101
19 MyorgDevelopmentEG2 0x0000000108f175de main + 190
20 libdyld.dylib 0x00007fff2025abbd start + 1
21 ??? 0x0000000000000005 0x0 + 5
)
<_ASDisplayLayer: 0x600007996580> => Waiting for UIView's draw/layout pass to complete. A draw/layout pass normally completes in the next runloop drain.
假设
这让我认为有一些孤立的子进程或守护进程仍然保留着资源。或者有工件没有被正确清理。
我尝试过的东西
- 关闭、擦除、删除构建之间的所有模拟器（无帮助）
xcrun simctl shutdown all
xcrun simctl erase all
xcrun simctl delete all
osascript -e 'tell application "iOS Simulator" to quit'
osascript -e 'tell application "Simulator" to quit'
- 在构建之间终止 CoreSimulator 相关服务（无帮助）
launchctl remove com.apple.CoreSimulator.CoreSimulatorService || true
launchctl remove com.apple.CoreSimulator.SimLaunchHost-x86 || true
launchctl remove com.apple.CoreSimulator.SimulatorTrampoline || true
launchctl remove com.apple.CoreSimulator.SimLaunchHost-arm64 || true
- 删除生成之间的模拟器设备、日志和缓存。这实际上有点帮助。薄片最初消失了，但在几次构建后很快又回来了，而机器仍在构建之间执行清洁....不确定为什么。
echo "Delete all simulator devices, cache, and temp"
rm -rf ~/Library/Developer/CoreSimulator
echo "Delete Simulator Logs"
rm -rf ~/Library/Logs/CoreSimulator
- 在 EarlGrey 上增加 Animation Timeout, Interaction Timeout
- Blacklist the URL 请求挂起。这确实通过忽略症状而不是修复实际问题来提高测试套件的稳定性，所以我希望我可以避免这种情况。
还有其他人 运行遇到同样的问题吗？感谢有人能对此有所了解或提出尝试建议。
注意：我们在 Buildkite CI 管道上使用 Bluepill and EarlGrey2 到 运行 UI 测试（iOS 14.5，Xcode 12.5）。
你试过了吗运行:
sudo systemsetup -setcomputersleep never