将 scipy 稀疏矩阵存储为 HDF5
Storing scipy sparse matrix as HDF5
我想以 HDF5 格式压缩和存储一个巨大的 Scipy 矩阵。我该怎么做呢？我试过下面的代码：
a = csr_matrix((dat, (row, col)), shape=(947969, 36039))
f = h5py.File('foo.h5','w')
dset = f.create_dataset("init", data=a, dtype = int, compression='gzip')
我遇到这样的错误，
TypeError: Scalar datasets don't support chunk/filter options
IOError: Can't prepare for writing data (No appropriate function for conversion path)
我无法将其转换为 numpy 数组，因为会出现内存溢出。什么是最好的方法？
你可以使用scipy.sparse.save_npz方法
或者考虑使用 Pandas.SparseDataFrame, but be aware that this method is very slow (thanks to )
演示：
正在生成稀疏矩阵和 SparseDataFrame
In [55]: import pandas as pd
In [56]: from scipy.sparse import *
In [57]: m = csr_matrix((20, 10), dtype=np.int8)
In [58]: m
Out[58]:
<20x10 sparse matrix of type '<class 'numpy.int8'>'
with 0 stored elements in Compressed Sparse Row format>
In [59]: sdf = pd.SparseDataFrame([pd.SparseSeries(m[i].toarray().ravel(), fill_value=0)
...: for i in np.arange(m.shape[0])])
...:
In [61]: type(sdf)
Out[61]: pandas.sparse.frame.SparseDataFrame
In [62]: sdf.info()
<class 'pandas.sparse.frame.SparseDataFrame'>
RangeIndex: 20 entries, 0 to 19
Data columns (total 10 columns):
0 20 non-null int8
1 20 non-null int8
2 20 non-null int8
3 20 non-null int8
4 20 non-null int8
5 20 non-null int8
6 20 non-null int8
7 20 non-null int8
8 20 non-null int8
9 20 non-null int8
dtypes: int8(10)
memory usage: 280.0 bytes
将 SparseDataFrame 保存到 HDF 文件
In [64]: sdf.to_hdf('d:/temp/sparse_df.h5', 'sparse_df')
从 HDF 文件读取
In [65]: store = pd.HDFStore('d:/temp/sparse_df.h5')
In [66]: store
Out[66]:
<class 'pandas.io.pytables.HDFStore'>
File path: d:/temp/sparse_df.h5
/sparse_df sparse_frame
In [67]: x = store['sparse_df']
In [68]: type(x)
Out[68]: pandas.sparse.frame.SparseDataFrame
In [69]: x.info()
<class 'pandas.sparse.frame.SparseDataFrame'>
Int64Index: 20 entries, 0 to 19
Data columns (total 10 columns):
0 20 non-null int8
1 20 non-null int8
2 20 non-null int8
3 20 non-null int8
4 20 non-null int8
5 20 non-null int8
6 20 non-null int8
7 20 non-null int8
8 20 non-null int8
9 20 non-null int8
dtypes: int8(10)
memory usage: 360.0 bytes
csr 矩阵将它的值存储在 3 个数组中。它不是数组或数组子类，所以h5py不能直接保存。你能做的最好的就是保存属性，并在加载时重新创建矩阵：
In [248]: M = sparse.random(5,10,.1, 'csr')
In [249]: M
Out[249]:
<5x10 sparse matrix of type '<class 'numpy.float64'>'
with 5 stored elements in Compressed Sparse Row format>
In [250]: M.data
Out[250]: array([ 0.91615298, 0.49907752, 0.09197862, 0.90442401, 0.93772772])
In [251]: M.indptr
Out[251]: array([0, 0, 1, 2, 3, 5], dtype=int32)
In [252]: M.indices
Out[252]: array([5, 7, 5, 2, 6], dtype=int32)
In [253]: M.data
Out[253]: array([ 0.91615298, 0.49907752, 0.09197862, 0.90442401, 0.93772772])
coo 格式具有 data、row、col 属性，与您用来创建 a 的 (dat, (row, col)) 基本相同].
In [254]: M.tocoo().row
Out[254]: array([1, 2, 3, 4, 4], dtype=int32)
新的 save_npz 功能：
arrays_dict = dict(format=matrix.format, shape=matrix.shape, data=matrix.data)
if matrix.format in ('csc', 'csr', 'bsr'):
arrays_dict.update(indices=matrix.indices, indptr=matrix.indptr)
...
elif matrix.format == 'coo':
arrays_dict.update(row=matrix.row, col=matrix.col)
...
np.savez(file, **arrays_dict)
换句话说，它将相关属性收集到字典中并使用 savez 创建 zip 存档。
同样的方法可以用于 h5py 文件。更多关于 save_npz 在最近的 SO 问题中，带有源代码链接。
看看你能不能让它工作。如果可以创建 csr 矩阵，则可以从其属性（或 coo 等价物）重新创建它。如果需要，我可以做一个工作示例。
csr 到 h5py 示例
import numpy as np
import h5py
from scipy import sparse
M = sparse.random(10,10,.2, 'csr')
print(repr(M))
print(M.data)
print(M.indices)
print(M.indptr)
f = h5py.File('sparse.h5','w')
g = f.create_group('Mcsr')
g.create_dataset('data',data=M.data)
g.create_dataset('indptr',data=M.indptr)
g.create_dataset('indices',data=M.indices)
g.attrs['shape'] = M.shape
f.close()
f = h5py.File('sparse.h5','r')
print(list(f.keys()))
print(list(f['Mcsr'].keys()))
g2 = f['Mcsr']
print(g2.attrs['shape'])
M1 = sparse.csr_matrix((g2['data'][:],g2['indices'][:],
g2['indptr'][:]), g2.attrs['shape'])
print(repr(M1))
print(np.allclose(M1.A, M.A))
f.close()
生产
1314:~/mypy$ python3 stack43390038.py
<10x10 sparse matrix of type '<class 'numpy.float64'>'
with 20 stored elements in Compressed Sparse Row format>
[ 0.13640389 0.92698959 .... 0.7762265 ]
[4 5 0 3 0 2 0 2 5 6 7 1 7 9 1 3 4 6 8 9]
[ 0 2 4 6 9 11 11 11 14 19 20]
['Mcsr']
['data', 'indices', 'indptr']
[10 10]
<10x10 sparse matrix of type '<class 'numpy.float64'>'
with 20 stored elements in Compressed Sparse Row format>
True
coo 替代品
Mo = M.tocoo()
g = f.create_group('Mcoo')
g.create_dataset('data', data=Mo.data)
g.create_dataset('row', data=Mo.row)
g.create_dataset('col', data=Mo.col)
g.attrs['shape'] = Mo.shape
g2 = f['Mcoo']
M2 = sparse.coo_matrix((g2['data'], (g2['row'], g2['col'])),
g2.attrs['shape']) # don't need the [:]
# could also use sparse.csr_matrix or M2.tocsr()
我想以 HDF5 格式压缩和存储一个巨大的 Scipy 矩阵。我该怎么做呢？我试过下面的代码：
a = csr_matrix((dat, (row, col)), shape=(947969, 36039))
f = h5py.File('foo.h5','w')
dset = f.create_dataset("init", data=a, dtype = int, compression='gzip')
我遇到这样的错误，
TypeError: Scalar datasets don't support chunk/filter options
IOError: Can't prepare for writing data (No appropriate function for conversion path)
我无法将其转换为 numpy 数组，因为会出现内存溢出。什么是最好的方法？
你可以使用scipy.sparse.save_npz方法
或者考虑使用 Pandas.SparseDataFrame, but be aware that this method is very slow (thanks to
演示：
正在生成稀疏矩阵和 SparseDataFrame
In [55]: import pandas as pd
In [56]: from scipy.sparse import *
In [57]: m = csr_matrix((20, 10), dtype=np.int8)
In [58]: m
Out[58]:
<20x10 sparse matrix of type '<class 'numpy.int8'>'
with 0 stored elements in Compressed Sparse Row format>
In [59]: sdf = pd.SparseDataFrame([pd.SparseSeries(m[i].toarray().ravel(), fill_value=0)
...: for i in np.arange(m.shape[0])])
...:
In [61]: type(sdf)
Out[61]: pandas.sparse.frame.SparseDataFrame
In [62]: sdf.info()
<class 'pandas.sparse.frame.SparseDataFrame'>
RangeIndex: 20 entries, 0 to 19
Data columns (total 10 columns):
0 20 non-null int8
1 20 non-null int8
2 20 non-null int8
3 20 non-null int8
4 20 non-null int8
5 20 non-null int8
6 20 non-null int8
7 20 non-null int8
8 20 non-null int8
9 20 non-null int8
dtypes: int8(10)
memory usage: 280.0 bytes
将 SparseDataFrame 保存到 HDF 文件
In [64]: sdf.to_hdf('d:/temp/sparse_df.h5', 'sparse_df')
从 HDF 文件读取
In [65]: store = pd.HDFStore('d:/temp/sparse_df.h5')
In [66]: store
Out[66]:
<class 'pandas.io.pytables.HDFStore'>
File path: d:/temp/sparse_df.h5
/sparse_df sparse_frame
In [67]: x = store['sparse_df']
In [68]: type(x)
Out[68]: pandas.sparse.frame.SparseDataFrame
In [69]: x.info()
<class 'pandas.sparse.frame.SparseDataFrame'>
Int64Index: 20 entries, 0 to 19
Data columns (total 10 columns):
0 20 non-null int8
1 20 non-null int8
2 20 non-null int8
3 20 non-null int8
4 20 non-null int8
5 20 non-null int8
6 20 non-null int8
7 20 non-null int8
8 20 non-null int8
9 20 non-null int8
dtypes: int8(10)
memory usage: 360.0 bytes
csr 矩阵将它的值存储在 3 个数组中。它不是数组或数组子类，所以h5py不能直接保存。你能做的最好的就是保存属性，并在加载时重新创建矩阵：
In [248]: M = sparse.random(5,10,.1, 'csr')
In [249]: M
Out[249]:
<5x10 sparse matrix of type '<class 'numpy.float64'>'
with 5 stored elements in Compressed Sparse Row format>
In [250]: M.data
Out[250]: array([ 0.91615298, 0.49907752, 0.09197862, 0.90442401, 0.93772772])
In [251]: M.indptr
Out[251]: array([0, 0, 1, 2, 3, 5], dtype=int32)
In [252]: M.indices
Out[252]: array([5, 7, 5, 2, 6], dtype=int32)
In [253]: M.data
Out[253]: array([ 0.91615298, 0.49907752, 0.09197862, 0.90442401, 0.93772772])
coo 格式具有 data、row、col 属性，与您用来创建 a 的 (dat, (row, col)) 基本相同].
In [254]: M.tocoo().row
Out[254]: array([1, 2, 3, 4, 4], dtype=int32)
新的 save_npz 功能：
arrays_dict = dict(format=matrix.format, shape=matrix.shape, data=matrix.data)
if matrix.format in ('csc', 'csr', 'bsr'):
arrays_dict.update(indices=matrix.indices, indptr=matrix.indptr)
...
elif matrix.format == 'coo':
arrays_dict.update(row=matrix.row, col=matrix.col)
...
np.savez(file, **arrays_dict)
换句话说，它将相关属性收集到字典中并使用 savez 创建 zip 存档。
同样的方法可以用于 h5py 文件。更多关于 save_npz 在最近的 SO 问题中，带有源代码链接。
看看你能不能让它工作。如果可以创建 csr 矩阵，则可以从其属性（或 coo 等价物）重新创建它。如果需要，我可以做一个工作示例。
csr 到 h5py 示例
import numpy as np
import h5py
from scipy import sparse
M = sparse.random(10,10,.2, 'csr')
print(repr(M))
print(M.data)
print(M.indices)
print(M.indptr)
f = h5py.File('sparse.h5','w')
g = f.create_group('Mcsr')
g.create_dataset('data',data=M.data)
g.create_dataset('indptr',data=M.indptr)
g.create_dataset('indices',data=M.indices)
g.attrs['shape'] = M.shape
f.close()
f = h5py.File('sparse.h5','r')
print(list(f.keys()))
print(list(f['Mcsr'].keys()))
g2 = f['Mcsr']
print(g2.attrs['shape'])
M1 = sparse.csr_matrix((g2['data'][:],g2['indices'][:],
g2['indptr'][:]), g2.attrs['shape'])
print(repr(M1))
print(np.allclose(M1.A, M.A))
f.close()
生产
1314:~/mypy$ python3 stack43390038.py
<10x10 sparse matrix of type '<class 'numpy.float64'>'
with 20 stored elements in Compressed Sparse Row format>
[ 0.13640389 0.92698959 .... 0.7762265 ]
[4 5 0 3 0 2 0 2 5 6 7 1 7 9 1 3 4 6 8 9]
[ 0 2 4 6 9 11 11 11 14 19 20]
['Mcsr']
['data', 'indices', 'indptr']
[10 10]
<10x10 sparse matrix of type '<class 'numpy.float64'>'
with 20 stored elements in Compressed Sparse Row format>
True
coo 替代品
Mo = M.tocoo()
g = f.create_group('Mcoo')
g.create_dataset('data', data=Mo.data)
g.create_dataset('row', data=Mo.row)
g.create_dataset('col', data=Mo.col)
g.attrs['shape'] = Mo.shape
g2 = f['Mcoo']
M2 = sparse.coo_matrix((g2['data'], (g2['row'], g2['col'])),
g2.attrs['shape']) # don't need the [:]
# could also use sparse.csr_matrix or M2.tocsr()