Caffe 在第一次迭代后崩溃？

Question

训练网络时 Caffe 崩溃。

在这种情况下，我只使用了 GPU 0。这是错误跟踪：

build/tools/caffe train -solver models/mv16f/solver.prototxt -gpu 0

I0113 14:21:05.861361 85242 solver.cpp:289] Learning Rate Policy: step
I0113 14:21:05.862876 85242 solver.cpp:341] Iteration 0, Testing net (#0)
I0113 14:21:30.271030 85242 solver.cpp:409]     Test net output #0: accuracy = 0.00872
I0113 14:21:30.271070 85242 solver.cpp:409]     Test net output #1: loss = 4.62895 (* 1 = 4.62895 loss)
I0113 14:21:32.317018 85242 solver.cpp:237] Iteration 0, loss = 4.62663
I0113 14:21:32.317062 85242 solver.cpp:253]     Train net output #0: loss = 4.62663 (* 1 = 4.62663 loss)
*** Aborted at 1452691298 (unix time) try "date -d @1452691298" if you are using GNU date ***
PC: @     0x7fe7f65f1cbc caffe::SGDSolver<>::GetLearningRate()
*** SIGFPE (@0x7fe7f65f1cbc) received by PID 85242 (TID 0x7fe7f72057c0) from PID 18446744073548012732; stack trace: ***
    @     0x7fe7f49c0d40 (unknown)
    @     0x7fe7f65f1cbc caffe::SGDSolver<>::GetLearningRate()
    @     0x7fe7f65f2281 caffe::SGDSolver<>::ApplyUpdate()
    @     0x7fe7f65d967c caffe::Solver<>::Step()
    @     0x7fe7f65d8990 caffe::Solver<>::Solve()
    @     0x7fe7f673251e caffe::P2PSync<>::run()
    @           0x416aa6 train()
    @           0x418c9a main
    @     0x7fe7f49abec5 (unknown)
    @           0x415819 (unknown)
@                0x0 (unknown)

火车的全部输出是here。

Answer 1

你的求解器文件有这一行

lr_policy: "fixed"

但是 Caffe 输出有这一行

lr_policy: "step"

如果使用逐步学习，则必须定义stepsize。同样，您的求解器文件表明您已经定义了步长，但 Caffe 输出显示没有步长。请再次检查您的解算器文件，并将此行添加回来

stepsize: 10000

Caffe 在第一次迭代后崩溃？

Caffe crashes after first iteration?

machine-learning

multi-gpu

ubuntu-14.04

caffe

pycaffe