Julia 无法在 docker 容器中加载大数据 [总线错误]

Question

我有一个加载 8 GB 数据的 Julia 程序。它在我的本地机器上运行良好。

但是当我在 docker 容器中尝试它时，它没有加载数据并给出总线错误。它可以很好地处理 docker 容器中 20 MB 的小数据。

Docker文件

FROM ubuntu:16.04

WORKDIR /julia

RUN apt-get -y update

RUN apt-get -y install unzip

RUN apt-get -y install cmake

RUN apt-get -y install clang

RUN apt-get -y install wget

RUN cd /tmp/

RUN wget "https://julialang.s3.amazonaws.com/bin/linux/x64/0.5/julia-0.5.0-linux-x86_64.tar.gz"

RUN tar -xzvf julia-0.5.0-linux-x86_64.tar.gz

RUN mv julia-3c9d75391c/ ~/julia

ENV PATH="/root/julia/bin:${PATH}"

RUN julia --eval 'Pkg.add("JSON")'

RUN julia --eval 'Pkg.add("HttpServer")'

RUN julia --eval 'Pkg.add("URIParser")'

RUN julia --eval 'Pkg.clone("https://github.com/deep-compute/AdaGram.jl.git")'

RUN julia --eval 'Pkg.build("AdaGram")'

CMD ["julia", "/tmp/adagram_server.jl", "80", "/julia/full.embed"]

docker-compose.yml

version: "3.1"
services:
    julia:
        image: ramidavalapati/julia:v-1
        volumes:
            - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed
        ports:
            - 8080:80
        command: ["sleep", "1h"]

接下来我在做

sudo docker-compose up
sudo docker exec -it $(sudo docker-compose ps -q julia) bash
root@3748d5958f77:/julia# julia
julia> using AdaGram
julia> AdaGram.load_model("/julia/full.embed")

错误

signal (7): Bus error
while loading no file, in expression starting on line 0
macro expansion at ./cartesian.jl:62 [inlined]
macro expansion at ./multidimensional.jl:429 [inlined]
_unsafe_batchsetindex! at ./multidimensional.jl:421
_setindex! at ./multidimensional.jl:370 [inlined]
setindex! at ./abstractarray.jl:832 [inlined]
#9 at /root/.julia/v0.5/AdaGram/src/AdaGram.jl:64
#600 at ./multi.jl:1030
run_work_thunk at ./multi.jl:1001
run_work_thunk at ./multi.jl:1010 [inlined]
#597 at ./event.jl:68
unknown function (ip: 0x7fe1822db16f)
jl_call_method_internal at /home/centos/buildbot/slave/package_tarball64/build/src/julia_internal.h:189 [inlined]
jl_apply_generic at /home/centos/buildbot/slave/package_tarball64/build/src/gf.c:1942
jl_apply at /home/centos/buildbot/slave/package_tarball64/build/src/julia.h:1392 [inlined]
start_task at /home/centos/buildbot/slave/package_tarball64/build/src/task.c:253
unknown function (ip: 0xffffffffffffffff)
Allocations: 9661042 (Pool: 9659980; Big: 1062); GC: 16
Bus error (core dumped)

Docker版本

Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:18 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:40:56 2017
 OS/Arch:      linux/amd64
 Experimental: false

Docker 信息

Containers: 24
 Running: 0
 Paused: 0
 Stopped: 24
Images: 24
Server Version: 17.09.0-ce
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
 NodeID: jlkmigmtyjhz6yvi3zuvkobu7
 Is Manager: true
 ClusterID: rqt03ulgvvnym235m1qm8vd17
 Managers: 4
 Nodes: 15
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
  Force Rotate: 0
 Autolock Managers: false
 Root Rotation In Progress: false
 Node Address: X.X.X.X
  Manager Addresses:
   X.X.X.X:2377
   X.X.X.X:2377
   X.X.X.X:2377
   X.X.X.X:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 06b9cb35161009dcb7123345749fef02f7cea8e0
 runc version: 3f2f8b84a77f73d38244dd690525642a72156c64
 init version: 949e6fa
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.10.0-35-generic
 Operating System: Ubuntu 16.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 251.8GiB
 Name: ram
 ID: 3OGG:275C:Q3IW:O4HX:DPLP:DPI3:5TIT:AG5J:EDMK:7NK3:L4UZ:BTQH
 Docker Root Dir: /var/lib/docker
 Debug Mode (client): false
 Debug Mode (server): false
 Username: ramidavalapati
 Registry: https://index.docker.io/v1/
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

 WARNING: No swap limit support

在此先感谢您的任何帮助

link to 40MBfull.embed file

Answer 1

问题是，docker 容器中没有足够的共享内存（默认为 64 MB）。

通过在运行宁docker图像时给出选项--shm-size解决了问题。

运行 : sudo docker 运行 --shm-size 1G sample:latest

docker-编写文件

version: "3.1"
services:
    julia:
        image: ramidavalapati/julia:v-1
        shm_size: 1g
        volumes:
            - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed
        ports:
            - 8080:80
        command: ["sleep", "1h"]

如果我们想在swarm mode中工作，我们需要引用卷部分中的共享内存。

version: "3.3"
services:
    julia:
        image: ramidavalapati/julia:v-1
        volumes:
            - /home/ram/adagram_data/40MBfull.embed:/julia/full.embed
            - /dev/shm:/dev/shm
        ports:
            - 8080:80
        command: ["sleep", "1h"]

此处容器使用所在主机的共享内存运行ning。

Julia 无法在 docker 容器中加载大数据 [总线错误]

Julia is not working for loading large data in docker container [Bus error]

julia

docker

docker-container