python-xarray：open_dataarray HPC 上的分段错误
python-xarray: open_dataarray Segmentation fault on HPC
我很难理解为什么在我的 Mac 上使用 open_dataarray in the HPC I use produces a Segmentation fault (core dumped). However, when I read in the file using open_dataarray 读取 NetCDF 文件效果很好。
从进一步调查来看，我创建的 NPac 文件（子部分）似乎有问题。以下是我生成文件的步骤：
- WaveWatchIII 输出一个文件
- 7 月提取：
$ncks -d time,0,30 in.nc out.nc 可以在 HPCout.nc 中打开 out.nc
- 提取变量：
$ncks -v hs in.nc out.nc 可以在 HPCout.nc 中打开 out.nc
- 提取域：
$ncks -d longitude,100.0,290.0 -d latitude,0.0,65.0 in.nc out.nc 无法在 HPC 上打开 out.nc 但可以在 Mac 上打开 out.nc。
这是我第一次看到这个问题，我相信这是由于 NetCDF 文件域的属性造成的。我猜它也可能与版本有关？我在 HPC 上完成了大部分繁重的工作，并使用我的 Mac 进行测试和理解，因此最好在 HPC 上运行它。
NetCDF文件可以下载here
ncdump -h ww3.Hs.July.NPac.nc
netcdf ww3.Hs.July.NPac {
dimensions:
time = UNLIMITED ; // (31 currently)
latitude = 66 ;
longitude = 191 ;
variables:
short hs(time, latitude, longitude) ;
hs:long_name = "significant height of wind and swell waves" ;
hs:standard_name = "sea_surface_wave_significant_height" ;
hs:globwave_name = "significant_wave_height" ;
hs:units = "m" ;
hs:_FillValue = -32767s ;
hs:scale_factor = 0.002f ;
hs:add_offset = 0.f ;
hs:valid_min = 0 ;
hs:valid_max = 32000 ;
float latitude(latitude) ;
latitude:units = "degree_north" ;
latitude:long_name = "latitude" ;
latitude:standard_name = "latitude" ;
latitude:valid_min = -90.f ;
latitude:valid_max = 90.f ;
latitude:axis = "Y" ;
float longitude(longitude) ;
longitude:units = "degree_east" ;
longitude:long_name = "longitude" ;
longitude:standard_name = "longitude" ;
longitude:valid_min = -180.f ;
longitude:valid_max = 180.f ;
longitude:axis = "X" ;
double time(time) ;
time:long_name = "julian day (UT)" ;
time:standard_name = "time" ;
time:units = "days since 1850-01-01T00:00:00Z" ;
time:conventions = "relative julian days with decimal part (as parts of the day )" ;
time:axis = "T" ;
// global attributes:
:WAVEWATCH_III_version_number = "4.18b" ;
:WAVEWATCH_III_switches = "NC4 F90 NOGRB NOPA LRB4 SHRD PR3 UQ FLX0 LN1 ST4 STAB0 NL1 BT1 DB1 MLIM TR0 BS0 IC0 REF0 XX0 WNT1 WNX1 CRT1 CRX1 O0 O1 O2 O3 O4 O5 O6 O7 O11 O14 TRKNC" ;
:SDS4\ namelist\ parameter\ WHITECAPWIDTH = 0.3f ;
:product_name = "ww3.199307.nc" ;
:area = "Indian Ocean Pacfic 1 degree" ;
:latitude_resolution = " 1.0000000" ;
:longitude_resolution = " 1.0000000" ;
:southernmost_latitude = "-70.0000000" ;
:northernmost_latitude = "65.0000000" ;
:westernmost_longitude = "20.0000000" ;
:easternmost_longitude = "295.0000000" ;
:minimum_altitude = "-12000 m" ;
:maximum_altitude = "9000 m" ;
:altitude_resolution = "n/a" ;
:start_date = "1993-07-01T00:00:00Z" ;
:stop_date = "1993-07-31T00:00:00Z" ;
:history = "Sat Dec 2 17:53:06 2017: ncks -O -d longitude,100.0,290.0 -d latitude,0.0,65.0 /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.Hs.July.nc /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.Hs.July.NPac.nc\nSat Dec 2 17:53:06 2017: ncks -O -v hs /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.July.nc /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.Hs.July.nc\nSat Dec 2 17:53:05 2017: ncks -O -d time,0,30 /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.19930701_19940630.nc /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.July.nc\nFri Nov 4 13:50:57 2016: ncrcat -O -o tmp.nc ww3.199307.nc ww3.199308.nc ww3.199309.nc ww3.199310.nc ww3.199311.nc ww3.199312.nc ww3.199401.nc ww3.199402.nc ww3.199403.nc ww3.199404.nc ww3.199405.nc ww3.199406.nc" ;
:nco_openmp_thread_number = 1 ;
:NCO = "4.3.7" ;
在 HPC 上：
$ nc-config --version
netCDF 4.2.1.1
>>> xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 2.6.32-431.el6.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
xarray: 0.10.0
pandas: 0.20.2
numpy: 1.13.1
scipy: 0.19.1
netCDF4: 1.2.4
h5netcdf: None
Nio: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.16.0
matplotlib: 2.0.2
cartopy: 0.15.1
seaborn: None
setuptools: 27.2.0
pip: 9.0.1
conda: 4.3.30
pytest: None
IPython: None
sphinx: None
>>> xr.open_dataarray('ww3.Hs.July.NPac.nc')
Segmentation fault (core dumped)
我的 Mac:
$ nc-config --version
netCDF 4.4.1
xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Darwin
OS-release: 17.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
xarray: 0.10.0
pandas: 0.20.1
numpy: 1.12.1
scipy: 1.0.0
netCDF4: 1.2.4
h5netcdf: None
Nio: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.16.0
matplotlib: 2.1.0
cartopy: 0.15.1
seaborn: 0.8.1
setuptools: 36.2.7
pip: 9.0.1
conda: 4.3.29
pytest: 3.2.0
IPython: 6.2.1
sphinx: 1.5.6
In [10]:xr.open_dataarray('ww3.Hs.July.NPac.nc')
Out[10]:
<xarray.DataArray 'hs' (time: 31, latitude: 66, longitude: 191)>
[390786 values with dtype=float64]
Coordinates:
* latitude (latitude) float32 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 ...
* longitude (longitude) float32 100.0 101.0 102.0 103.0 104.0 105.0 106.0 ...
* time (time) datetime64[ns] 2013-07-01 2013-07-02 2013-07-03 ...
Attributes:
long_name: significant height of wind and swell waves
standard_name: sea_surface_wave_significant_height
globwave_name: significant_wave_height
units: m
valid_min: 0
valid_max: 32000
抱歉。看起来我的 ncks 命令没有完成 属性。与 xarray 无关，只是奇怪我的 Mac 没有抛出错误。奇怪的是，给 out.nc 一个不同的文件名似乎可以解决问题
$ncks -O -d longitude,100.0,290.0 -d latitude,0.0,65.0 ww3.Hs.July.nc ww3.Hs.July.NPac.nc
$python
>>>import xarray as xr
>>>xr.open_dataarray('ww3.Hs.July.NPac.nc')
Segmentation fault (core dumped)
$ncks -O -d longitude,100.0,290.0 -d latitude,0.0,65.0 ww3.Hs.July.nc ww3.Hs.July.NPac1.nc
$python
>>>import xarray as xr
>>>xr.open_dataarray('ww3.Hs.July.NPac1.nc')
<xarray.DataArray 'hs' (time: 31, latitude: 66, longitude: 191)>
[390786 values with dtype=float64]
Coordinates:
* latitude (latitude) float32 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 ...
* longitude (longitude) float32 100.0 101.0 102.0 103.0 104.0 105.0 106.0 ...
* time (time) datetime64[ns] 1993-07-01 1993-07-02 1993-07-03 ...
Attributes:
long_name: significant height of wind and swell waves
standard_name: sea_surface_wave_significant_height
globwave_name: significant_wave_height
units: m
valid_min: 0
valid_max: 32000
我很难理解为什么在我的 Mac 上使用 open_dataarray in the HPC I use produces a Segmentation fault (core dumped). However, when I read in the file using open_dataarray 读取 NetCDF 文件效果很好。
从进一步调查来看，我创建的 NPac 文件（子部分）似乎有问题。以下是我生成文件的步骤：
- WaveWatchIII 输出一个文件
- 7 月提取：
$ncks -d time,0,30 in.nc out.nc可以在 HPCout.nc 中打开 out.nc - 提取变量：
$ncks -v hs in.nc out.nc可以在 HPCout.nc 中打开 out.nc - 提取域：
$ncks -d longitude,100.0,290.0 -d latitude,0.0,65.0 in.nc out.nc无法在 HPC 上打开 out.nc 但可以在 Mac 上打开 out.nc。
这是我第一次看到这个问题，我相信这是由于 NetCDF 文件域的属性造成的。我猜它也可能与版本有关？我在 HPC 上完成了大部分繁重的工作，并使用我的 Mac 进行测试和理解，因此最好在 HPC 上运行它。
NetCDF文件可以下载here
ncdump -h ww3.Hs.July.NPac.nc
netcdf ww3.Hs.July.NPac {
dimensions:
time = UNLIMITED ; // (31 currently)
latitude = 66 ;
longitude = 191 ;
variables:
short hs(time, latitude, longitude) ;
hs:long_name = "significant height of wind and swell waves" ;
hs:standard_name = "sea_surface_wave_significant_height" ;
hs:globwave_name = "significant_wave_height" ;
hs:units = "m" ;
hs:_FillValue = -32767s ;
hs:scale_factor = 0.002f ;
hs:add_offset = 0.f ;
hs:valid_min = 0 ;
hs:valid_max = 32000 ;
float latitude(latitude) ;
latitude:units = "degree_north" ;
latitude:long_name = "latitude" ;
latitude:standard_name = "latitude" ;
latitude:valid_min = -90.f ;
latitude:valid_max = 90.f ;
latitude:axis = "Y" ;
float longitude(longitude) ;
longitude:units = "degree_east" ;
longitude:long_name = "longitude" ;
longitude:standard_name = "longitude" ;
longitude:valid_min = -180.f ;
longitude:valid_max = 180.f ;
longitude:axis = "X" ;
double time(time) ;
time:long_name = "julian day (UT)" ;
time:standard_name = "time" ;
time:units = "days since 1850-01-01T00:00:00Z" ;
time:conventions = "relative julian days with decimal part (as parts of the day )" ;
time:axis = "T" ;
// global attributes:
:WAVEWATCH_III_version_number = "4.18b" ;
:WAVEWATCH_III_switches = "NC4 F90 NOGRB NOPA LRB4 SHRD PR3 UQ FLX0 LN1 ST4 STAB0 NL1 BT1 DB1 MLIM TR0 BS0 IC0 REF0 XX0 WNT1 WNX1 CRT1 CRX1 O0 O1 O2 O3 O4 O5 O6 O7 O11 O14 TRKNC" ;
:SDS4\ namelist\ parameter\ WHITECAPWIDTH = 0.3f ;
:product_name = "ww3.199307.nc" ;
:area = "Indian Ocean Pacfic 1 degree" ;
:latitude_resolution = " 1.0000000" ;
:longitude_resolution = " 1.0000000" ;
:southernmost_latitude = "-70.0000000" ;
:northernmost_latitude = "65.0000000" ;
:westernmost_longitude = "20.0000000" ;
:easternmost_longitude = "295.0000000" ;
:minimum_altitude = "-12000 m" ;
:maximum_altitude = "9000 m" ;
:altitude_resolution = "n/a" ;
:start_date = "1993-07-01T00:00:00Z" ;
:stop_date = "1993-07-31T00:00:00Z" ;
:history = "Sat Dec 2 17:53:06 2017: ncks -O -d longitude,100.0,290.0 -d latitude,0.0,65.0 /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.Hs.July.nc /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.Hs.July.NPac.nc\nSat Dec 2 17:53:06 2017: ncks -O -v hs /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.July.nc /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.Hs.July.nc\nSat Dec 2 17:53:05 2017: ncks -O -d time,0,30 /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.19930701_19940630.nc /projects/rsmas/kirtman/rxb826/WW3exps/IO_Pac_CCSM4/CCSM4_19930701_19940630_1a1/work/ww3.July.nc\nFri Nov 4 13:50:57 2016: ncrcat -O -o tmp.nc ww3.199307.nc ww3.199308.nc ww3.199309.nc ww3.199310.nc ww3.199311.nc ww3.199312.nc ww3.199401.nc ww3.199402.nc ww3.199403.nc ww3.199404.nc ww3.199405.nc ww3.199406.nc" ;
:nco_openmp_thread_number = 1 ;
:NCO = "4.3.7" ;
在 HPC 上：
$ nc-config --version
netCDF 4.2.1.1
>>> xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Linux
OS-release: 2.6.32-431.el6.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
xarray: 0.10.0
pandas: 0.20.2
numpy: 1.13.1
scipy: 0.19.1
netCDF4: 1.2.4
h5netcdf: None
Nio: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.16.0
matplotlib: 2.0.2
cartopy: 0.15.1
seaborn: None
setuptools: 27.2.0
pip: 9.0.1
conda: 4.3.30
pytest: None
IPython: None
sphinx: None
>>> xr.open_dataarray('ww3.Hs.July.NPac.nc')
Segmentation fault (core dumped)
我的 Mac:
$ nc-config --version
netCDF 4.4.1
xr.show_versions()
INSTALLED VERSIONS
------------------
commit: None
python: 3.6.3.final.0
python-bits: 64
OS: Darwin
OS-release: 17.2.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: en_US.UTF-8
LANG: en_US.UTF-8
LOCALE: en_US.UTF-8
xarray: 0.10.0
pandas: 0.20.1
numpy: 1.12.1
scipy: 1.0.0
netCDF4: 1.2.4
h5netcdf: None
Nio: None
bottleneck: 1.2.1
cyordereddict: None
dask: 0.16.0
matplotlib: 2.1.0
cartopy: 0.15.1
seaborn: 0.8.1
setuptools: 36.2.7
pip: 9.0.1
conda: 4.3.29
pytest: 3.2.0
IPython: 6.2.1
sphinx: 1.5.6
In [10]:xr.open_dataarray('ww3.Hs.July.NPac.nc')
Out[10]:
<xarray.DataArray 'hs' (time: 31, latitude: 66, longitude: 191)>
[390786 values with dtype=float64]
Coordinates:
* latitude (latitude) float32 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 ...
* longitude (longitude) float32 100.0 101.0 102.0 103.0 104.0 105.0 106.0 ...
* time (time) datetime64[ns] 2013-07-01 2013-07-02 2013-07-03 ...
Attributes:
long_name: significant height of wind and swell waves
standard_name: sea_surface_wave_significant_height
globwave_name: significant_wave_height
units: m
valid_min: 0
valid_max: 32000
抱歉。看起来我的 ncks 命令没有完成 属性。与 xarray 无关，只是奇怪我的 Mac 没有抛出错误。奇怪的是，给 out.nc 一个不同的文件名似乎可以解决问题
$ncks -O -d longitude,100.0,290.0 -d latitude,0.0,65.0 ww3.Hs.July.nc ww3.Hs.July.NPac.nc
$python
>>>import xarray as xr
>>>xr.open_dataarray('ww3.Hs.July.NPac.nc')
Segmentation fault (core dumped)
$ncks -O -d longitude,100.0,290.0 -d latitude,0.0,65.0 ww3.Hs.July.nc ww3.Hs.July.NPac1.nc
$python
>>>import xarray as xr
>>>xr.open_dataarray('ww3.Hs.July.NPac1.nc')
<xarray.DataArray 'hs' (time: 31, latitude: 66, longitude: 191)>
[390786 values with dtype=float64]
Coordinates:
* latitude (latitude) float32 0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 ...
* longitude (longitude) float32 100.0 101.0 102.0 103.0 104.0 105.0 106.0 ...
* time (time) datetime64[ns] 1993-07-01 1993-07-02 1993-07-03 ...
Attributes:
long_name: significant height of wind and swell waves
standard_name: sea_surface_wave_significant_height
globwave_name: significant_wave_height
units: m
valid_min: 0
valid_max: 32000