在 macOS 上使用 OpenMP 使用 Cython 进行编译

Question

我在 macOS Mojave 10.14.6 上，我正在尝试从 this 存储库中用 c 和 c++ 编译一些必需的扩展模块：

python setup.py build_ext --inplace

这给了我以下错误：

No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
running build_ext
/Users/user/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py:249: UserWarning: 

                               !! WARNING !!

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Your compiler (g++) is not compatible with the compiler Pytorch was
built with for this platform, which is clang++ on darwin. Please
use clang++ to to compile your extension. Alternatively, you may
compile PyTorch from source using g++, and then you can also use
g++ to compile your extension.

See https://github.com/pytorch/pytorch/blob/master/CONTRIBUTING.md for help
with compiling PyTorch from source.
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

                              !! WARNING !!

  warnings.warn(WRONG_COMPILER_WARNING.format(

再往下：

clang: error: unknown argument: '-i'
clang: error: no such file or directory: 'sysroot'
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang: error: unsupported option '-fopenmp'
ninja: build stopped: subcommand failed.

（这应该是相关部分，最近的回溯也显示 subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1. 最后是 RuntimeError: Error compiling objects for extension）

根据我的理解和目前的尝试：

设置文件中的标志 -fopenmp 是问题之一，似乎缺少 OpenMP
因此我安装了 brew install llvm 和 brew install libomp
将 PATH=\"/usr/local/bin:$PATH\ 添加到我的 .bash_profile 希望现在可以使用新安装的编译器
根据here在设置文件-fopenmp之前添加标志-Xpreprocessor，这样extra_compile_args=['-Xpreprocessor', '-std=c99', '-O3', '-fopenmp']
我尝试了 CC=gcc python setup.py build_ext --inplace，因为关于将 'c' 视为 'c++' 的警告，但由于我正在尝试构建 C 和 C++ 模块，我不确定是什么变量实际上应该设置为
我也按照建议尝试了 CC=/usr/local/opt/llvm/bin/clang++ python setup.py build_ext --inplace here
尝试在没有 -fopenmp 的情况下构建它（这实际上不是一个选项，因为我需要 OpenMP IIRC 提供的并行执行）基本上导致相同的错误，但没有关于 OpenMP[=71 的错误=]
我还需要设置 CXXFLAGS 和 CFLAGS 吗？
现在我正在 conda 环境中完成所有这些工作，Anaconda 似乎自带了自己的工具。我应该使用那些吗？需要什么 MacOSX SDK？

对于这个令人困惑的问题，我深表歉意，但我对编译器没有先验知识并试图解决这个问题就像掉进了兔子洞......我将非常感谢任何对这种情况有所了解的评论我很乐意提供任何遗漏的信息！

更新：

清理 .bash_profile 并导出 export CC='gcc-10'; export CXX='clang++' 消除了 Pytorch 警告，但 clang 问题仍然存在
仍然与 os.environ['CC'] = 'gcc-10' 和 os.environ['CXX'] = 'clang++ 在 setup.py

完整的错误信息是：

(torch) [10:48:05] vanessamac: occupancy_networks $ python setup.py build_ext --inplace --verbose
No CUDA runtime is found, using CUDA_HOME='/usr/local/cuda'
running build_ext
building 'im2mesh.utils.libkdtree.pykdtree.kdtree' extension
Emitting ninja build file /Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/build.ninja...
Compiling objects...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/2] clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
FAILED: /Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o 
clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/_kdtree_core.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang: error: unsupported option '-fopenmp'
[2/2] clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/kdtree.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
FAILED: /Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o 
clang++ -MMD -MF '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o'.d -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/include -arch x86_64 -I/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/numpy/core/include -I/Users/vanessamac/miniconda3/envs/torch/include/python3.8 -c -c '/Users/vanessamac/projects/occupancy_networks/im2mesh/utils/libkdtree/pykdtree/kdtree.c' -o '/Users/vanessamac/projects/occupancy_networks/build/temp.macosx-10.9-x86_64-3.8/im2mesh/utils/libkdtree/pykdtree/kdtree.o' -std=c99 -O3 -fopenmp -DTORCH_API_INCLUDE_EXTENSION_H -DTORCH_EXTENSION_NAME=kdtree -D_GLIBCXX_USE_CXX11_ABI=0
clang: warning: treating 'c' input as 'c++' when in C++ mode, this behavior is deprecated [-Wdeprecated]
clang: error: unsupported option '-fopenmp'
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1509, in _run_ninja_build
    subprocess.run(
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/subprocess.py", line 512, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "setup.py", line 112, in <module>
    setup(
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/setuptools/__init__.py", line 165, in setup
    return distutils.core.setup(**attrs)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 87, in run
    _build_ext.run(self)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 649, in build_extensions
    build_ext.build_extensions(self)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/setuptools/command/build_ext.py", line 208, in build_extension
    _build_ext.build_extension(self, ext)
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/distutils/command/build_ext.py", line 528, in build_extension
    objects = self.compiler.compile(sources,
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 469, in unix_wrap_ninja_compile
    _write_ninja_file_and_compile_objects(
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1228, in _write_ninja_file_and_compile_objects
    _run_ninja_build(
  File "/Users/vanessamac/miniconda3/envs/torch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 1529, in _run_ninja_build
    raise RuntimeError(message)
RuntimeError: Error compiling objects for extension

Answer 1

这里有一些提示：

在 macOS 上使用 gcc 而不是 llvm 或 clang 无痛 openmp-support。请注意，Apple 的默认 gcc 只是 Apple clang 的别名，您将在 gcc --version 中看到。你可以安装 真正的 gcc 和自制软件：brew install gcc.
然后在同一个终端window中使用export CC='gcc-10'（最新版本应该是gcc 10.x）暂时使用homebrew的gcc作为你的C编译器。
不需要设置CXXFLAGS或CFLAGS。所需的标志由 setup.py.
中的 distutils/setuptools 设置
您将无法在 macOS 10.14.6 上编译 dmc_cuda_module。 nvidia 提供的最新 macOS 版本 cuda 驱动程序是 10.13.6。因此，您可以取消注释 setup.py 的这一部分，并希望最好不要使用此模块...
setup.py 中的某些扩展在使用 numpy C-API 时不包括 numpy headers。在 macOS 上，有必要为每个扩展包含 numpy headers，请参阅 this comment。因此，您必须将 include_dirs=[numpy_include_dir] 添加到这些扩展中。
编辑：正如聊天中所讨论的：错误是由于 conda env 忽略了 CC 变量。通过自制软件安装 python+pip 并通过 pip 安装所需的 python 软件包后，此答案的步骤适用于 OP。

总而言之，这是对我有用的 setup.py（macOS 10.5.7、gcc-10）：

try:
    from setuptools import setup
except ImportError:
    from distutils.core import setup
from distutils.extension import Extension
from Cython.Build import cythonize
from torch.utils.cpp_extension import BuildExtension, CppExtension, CUDAExtension
import numpy


# Get the numpy include directory.
numpy_include_dir = numpy.get_include()

# Extensions
# pykdtree (kd tree)
pykdtree = Extension(
    'im2mesh.utils.libkdtree.pykdtree.kdtree',
    sources=[
        'im2mesh/utils/libkdtree/pykdtree/kdtree.c',
        'im2mesh/utils/libkdtree/pykdtree/_kdtree_core.c'
    ],
    language='c',
    extra_compile_args=['-std=c99', '-O3', '-fopenmp'],
    extra_link_args=['-lgomp'],
    include_dirs=[numpy_include_dir]
)

# mcubes (marching cubes algorithm)
mcubes_module = Extension(
    'im2mesh.utils.libmcubes.mcubes',
    sources=[
        'im2mesh/utils/libmcubes/mcubes.pyx',
        'im2mesh/utils/libmcubes/pywrapper.cpp',
        'im2mesh/utils/libmcubes/marchingcubes.cpp'
    ],
    language='c++',
    extra_compile_args=['-std=c++11'],
    include_dirs=[numpy_include_dir]
)

# triangle hash (efficient mesh intersection)
triangle_hash_module = Extension(
    'im2mesh.utils.libmesh.triangle_hash',
    sources=[
        'im2mesh/utils/libmesh/triangle_hash.pyx'
    ],
    libraries=['m'],  # Unix-like specific
    include_dirs=[numpy_include_dir]
)

# mise (efficient mesh extraction)
mise_module = Extension(
    'im2mesh.utils.libmise.mise',
    sources=[
        'im2mesh/utils/libmise/mise.pyx'
    ],
)

# simplify (efficient mesh simplification)
simplify_mesh_module = Extension(
    'im2mesh.utils.libsimplify.simplify_mesh',
    sources=[
        'im2mesh/utils/libsimplify/simplify_mesh.pyx'
    ],
    include_dirs=[numpy_include_dir]
)

# voxelization (efficient mesh voxelization)
voxelize_module = Extension(
    'im2mesh.utils.libvoxelize.voxelize',
    sources=[
        'im2mesh/utils/libvoxelize/voxelize.pyx'
    ],
    libraries=['m']  # Unix-like specific
)

# DMC extensions
dmc_pred2mesh_module = CppExtension(
    'im2mesh.dmc.ops.cpp_modules.pred2mesh',
    sources=[
        'im2mesh/dmc/ops/cpp_modules/pred_to_mesh_.cpp',
    ]   
)

# dmc_cuda_module = CUDAExtension(
#     'im2mesh.dmc.ops._cuda_ext', 
#     sources=[
#         'im2mesh/dmc/ops/src/extension.cpp',
#         'im2mesh/dmc/ops/src/curvature_constraint_kernel.cu',
#         'im2mesh/dmc/ops/src/grid_pooling_kernel.cu',
#         'im2mesh/dmc/ops/src/occupancy_to_topology_kernel.cu',
#         'im2mesh/dmc/ops/src/occupancy_connectivity_kernel.cu',
#         'im2mesh/dmc/ops/src/point_triangle_distance_kernel.cu',
#     ]
# )

# Gather all extension modules
ext_modules = [
    pykdtree,
    mcubes_module,
    triangle_hash_module,
    mise_module,
    simplify_mesh_module,
    voxelize_module,
    dmc_pred2mesh_module,
    #dmc_cuda_module,
]

setup(
    ext_modules=cythonize(ext_modules),
    cmdclass={
        'build_ext': BuildExtension
    }
)

在 macOS 上使用 OpenMP 使用 Cython 进行编译

Compiling with Cython using OpenMP on macOS

c++

python

macos

cython