全球/本地环境影响 Haskell 的标准基准测试结果
Global / local environment affects Haskell's Criterion benchmarks results
我们正在对我们公司的一些 Haskell 代码进行基准测试,我们刚刚遇到了一个非常奇怪的案例。这是一个代码,它对同一件事进行了两次基准测试。前者使用 Criterion.env
为所有测试创建一次,后者为每个测试创建 env。这是唯一的区别,但是为每个工作台创建 env 的那个运行速度快 5 倍。
有谁知道是什么原因造成的?最小示例:
module Main where
import Prelude
import Control.Monad
import qualified Data.Vector.Storable.Mutable as Vector
import qualified Data.Vector.Storable as Vector
import Data.Vector.Storable (Vector)
import Criterion.Main
testf :: Int -> Vector Int -> IO (Vector Int)
testf !i !v = do
mv <- Vector.unsafeThaw v
let go j = do
x <- if j == 0 then return 0 else Vector.unsafeRead mv (j - 1)
Vector.unsafeWrite mv j (x+1)
when (j < i - 1) $ go (j + 1)
go 0
Vector.unsafeFreeze mv
mkVec :: Int -> IO (Vector Int)
mkVec !i = Vector.unsafeFreeze =<< Vector.new (i + 1)
main :: IO ()
main = do
defaultMain
[ bgroup "env per all runs"
$ (\(i :: Int) -> env (mkVec (10 ^ i))
$ \v -> bench ("10e" ++ show i)
$ nfIO (testf (10 ^ i) v)) <$> [7..8]
, bgroup "env per every run"
$ (\(i :: Int) -> bench ("10e" ++ show i)
$ perRunEnv (mkVec (10 ^ i))
$ (testf (10 ^ i))) <$> [7..8]
]
结果:
benchmarking env per all runs/10e7
time 17.34 ms (17.20 ms .. 17.41 ms)
0.999 R² (0.998 R² .. 1.000 R²)
mean 17.43 ms (17.34 ms .. 17.67 ms)
std dev 321.5 μs (142.1 μs .. 591.3 μs)
benchmarking env per all runs/10e8
time 173.5 ms (173.2 ms .. 173.8 ms)
1.000 R² (1.000 R² .. 1.000 R²)
mean 173.8 ms (173.6 ms .. 174.0 ms)
std dev 279.5 μs (194.9 μs .. 355.6 μs)
variance introduced by outliers: 12% (moderately inflated)
benchmarking env per every run/10e7
time 4.289 ms (1.807 ms .. 5.771 ms)
0.924 R² (0.696 R² .. 1.000 R²)
mean 8.903 ms (5.752 ms .. 14.20 ms)
std dev 5.029 ms (249.0 μs .. 6.244 ms)
variance introduced by outliers: 79% (severely inflated)
benchmarking env per every run/10e8
time 53.76 ms (30.23 ms .. 98.51 ms)
0.940 R² (0.920 R² .. 1.000 R²)
mean 102.9 ms (68.67 ms .. 127.1 ms)
std dev 36.55 ms (0.0 s .. 41.99 ms)
variance introduced by outliers: 73% (severely inflated)
关于你可能是同事的问题,我无法重现这个问题。注意我使用的是 GHC 8.2.2 和标准 1.3.0.0
benchmarking env per all runs/10e7
time 18.94 ms (18.71 ms .. 19.22 ms)
0.999 R² (0.998 R² .. 1.000 R²)
mean 19.59 ms (19.39 ms .. 19.99 ms)
std dev 618.3 μs (379.7 μs .. 952.8 μs)
benchmarking env per all runs/10e8
time 192.0 ms (189.5 ms .. 194.9 ms)
1.000 R² (0.999 R² .. 1.000 R²)
mean 191.8 ms (190.3 ms .. 193.1 ms)
std dev 1.778 ms (1.088 ms .. 2.457 ms)
variance introduced by outliers: 14% (moderately inflated)
benchmarking env per every run/10e7
time 18.97 ms (18.38 ms .. 19.62 ms)
0.999 R² (0.996 R² .. 1.000 R²)
mean 18.98 ms (18.83 ms .. 19.35 ms)
std dev 298.8 μs (25.39 μs .. 391.3 μs)
variance introduced by outliers: 14% (moderately inflated)
benchmarking env per every run/10e8
time 194.0 ms (182.0 ms .. 211.6 ms)
0.999 R² (0.997 R² .. 1.000 R²)
mean 192.0 ms (189.0 ms .. 193.9 ms)
std dev 2.850 ms (0.0 s .. 3.261 ms)
variance introduced by outliers: 19% (moderately inflated)
./p 18.32s user 1.16s system 99% cpu 19.531 total
% ghc-pkg list criterion
...
criterion-1.3.0.0
我注意到我的标准版本是在@jberryman 指出的错误被修复后上传的,所以也许这就是不同之处。
我们正在对我们公司的一些 Haskell 代码进行基准测试,我们刚刚遇到了一个非常奇怪的案例。这是一个代码,它对同一件事进行了两次基准测试。前者使用 Criterion.env
为所有测试创建一次,后者为每个测试创建 env。这是唯一的区别,但是为每个工作台创建 env 的那个运行速度快 5 倍。
有谁知道是什么原因造成的?最小示例:
module Main where
import Prelude
import Control.Monad
import qualified Data.Vector.Storable.Mutable as Vector
import qualified Data.Vector.Storable as Vector
import Data.Vector.Storable (Vector)
import Criterion.Main
testf :: Int -> Vector Int -> IO (Vector Int)
testf !i !v = do
mv <- Vector.unsafeThaw v
let go j = do
x <- if j == 0 then return 0 else Vector.unsafeRead mv (j - 1)
Vector.unsafeWrite mv j (x+1)
when (j < i - 1) $ go (j + 1)
go 0
Vector.unsafeFreeze mv
mkVec :: Int -> IO (Vector Int)
mkVec !i = Vector.unsafeFreeze =<< Vector.new (i + 1)
main :: IO ()
main = do
defaultMain
[ bgroup "env per all runs"
$ (\(i :: Int) -> env (mkVec (10 ^ i))
$ \v -> bench ("10e" ++ show i)
$ nfIO (testf (10 ^ i) v)) <$> [7..8]
, bgroup "env per every run"
$ (\(i :: Int) -> bench ("10e" ++ show i)
$ perRunEnv (mkVec (10 ^ i))
$ (testf (10 ^ i))) <$> [7..8]
]
结果:
benchmarking env per all runs/10e7
time 17.34 ms (17.20 ms .. 17.41 ms)
0.999 R² (0.998 R² .. 1.000 R²)
mean 17.43 ms (17.34 ms .. 17.67 ms)
std dev 321.5 μs (142.1 μs .. 591.3 μs)
benchmarking env per all runs/10e8
time 173.5 ms (173.2 ms .. 173.8 ms)
1.000 R² (1.000 R² .. 1.000 R²)
mean 173.8 ms (173.6 ms .. 174.0 ms)
std dev 279.5 μs (194.9 μs .. 355.6 μs)
variance introduced by outliers: 12% (moderately inflated)
benchmarking env per every run/10e7
time 4.289 ms (1.807 ms .. 5.771 ms)
0.924 R² (0.696 R² .. 1.000 R²)
mean 8.903 ms (5.752 ms .. 14.20 ms)
std dev 5.029 ms (249.0 μs .. 6.244 ms)
variance introduced by outliers: 79% (severely inflated)
benchmarking env per every run/10e8
time 53.76 ms (30.23 ms .. 98.51 ms)
0.940 R² (0.920 R² .. 1.000 R²)
mean 102.9 ms (68.67 ms .. 127.1 ms)
std dev 36.55 ms (0.0 s .. 41.99 ms)
variance introduced by outliers: 73% (severely inflated)
关于你可能是同事的问题,我无法重现这个问题。注意我使用的是 GHC 8.2.2 和标准 1.3.0.0
benchmarking env per all runs/10e7
time 18.94 ms (18.71 ms .. 19.22 ms)
0.999 R² (0.998 R² .. 1.000 R²)
mean 19.59 ms (19.39 ms .. 19.99 ms)
std dev 618.3 μs (379.7 μs .. 952.8 μs)
benchmarking env per all runs/10e8
time 192.0 ms (189.5 ms .. 194.9 ms)
1.000 R² (0.999 R² .. 1.000 R²)
mean 191.8 ms (190.3 ms .. 193.1 ms)
std dev 1.778 ms (1.088 ms .. 2.457 ms)
variance introduced by outliers: 14% (moderately inflated)
benchmarking env per every run/10e7
time 18.97 ms (18.38 ms .. 19.62 ms)
0.999 R² (0.996 R² .. 1.000 R²)
mean 18.98 ms (18.83 ms .. 19.35 ms)
std dev 298.8 μs (25.39 μs .. 391.3 μs)
variance introduced by outliers: 14% (moderately inflated)
benchmarking env per every run/10e8
time 194.0 ms (182.0 ms .. 211.6 ms)
0.999 R² (0.997 R² .. 1.000 R²)
mean 192.0 ms (189.0 ms .. 193.9 ms)
std dev 2.850 ms (0.0 s .. 3.261 ms)
variance introduced by outliers: 19% (moderately inflated)
./p 18.32s user 1.16s system 99% cpu 19.531 total
% ghc-pkg list criterion
...
criterion-1.3.0.0
我注意到我的标准版本是在@jberryman 指出的错误被修复后上传的,所以也许这就是不同之处。