Python

Question

想知道是否有更有效的方法将文件内容加载到稀疏矩阵中。下面的代码从一个大文件 (8GB) 中读取数据，该文件大部分为零值（非常稀疏），然后对读取的每一行进行一些处理。我想有效地对其执行算术运算，所以我尝试将这些线存储为稀疏矩阵。由于事先不知道文件中的行数，并且 array/matrix 不是动态的，我必须先将其存储在列表中，然后转换为 csr_matrix。此阶段 ("X = csr_matrix(X)") 占用大量时间和内存。
有什么建议吗？

import numpy as np
from scipy.sparse import csr_matrix
from datetime import datetime as time

global header_names; header_names = []

def readOppFromFile(filepath):

    print "Read Opportunities From File..." + str(time.now())

    # read file header - feature names separated with commas
    global header_names

    with open(filepath, "r") as f:

        i=0

        header_names  = f.readline().rstrip().split(',')

        for line in f: 


            # replace empty string with 0 in comma-separated string. In addition, clean null values (replace with 0)
            yield [(x.replace('null', '0') if x else 0) for x in line.rstrip().split(',')]
            i += 1

        print "Number of opportunities read from file: %s" % str(i) 

def processOpportunities(opp_data):

    print "Process Opportunities ..." + str(time.now())

    # Initialization 
    X = []
    targets_array = []

    global header_names

    for opportunity in opp_data:

        # Extract for each opportunity it's target variable, save it in a special array and then remove it  
        target = opportunity[-1] # Only last column
        targets_array.append(target)
        del opportunity[-1] # Remove last column

        X.append(opportunity)     

   print " Starting to transform to a sparse matrix" + str(time.now())
    X = csr_matrix(X)
    print "Finished transform to a sparse matrix " + str(time.now())

    # The target variable of each impression
    targets_array = np.array(targets_array, dtype=int)
    print "targets_array" + str(time.now())        

    return X, targets_array

def main():


    print "STRAT -----> " + str(time.now())
    running_time = time.now()

    opps_data = readOppFromFile(inputfilename)

    features, target = processOpportunities(opps_data)

if __name__ == '__main__':

    """ ################### GLOBAL VARIABLES ############################ """     
    inputfilename = 'C:/somefolder/trainingset.working.csv'

    """ ################### START PROGRAM ############################ """     
    main()

更新： 矩阵的维度不是恒定的，它们取决于输入文件，并且可能在程序的每个运行中发生变化。有关我输入的一小部分示例，请参阅 here。

Answer 1

如果您手动跟踪非零值，您可以直接构造一个稀疏矩阵：

X_data = []
X_row, X_col = [], []
targets_array = []

for row_idx, opportunity in enumerate(opp_data):
    targets_array.append(int(opportunity[-1]))
    row = np.array(map(int, opportunity[:-1]))
    col_inds, = np.nonzero(row)
    X_col.extend(col_inds)
    X_row.extend([row_idx]*len(col_inds))
    X_data.extend(row[col_inds])

print " Starting to transform to a sparse matrix" + str(time.now())
X = coo_matrix((X_data, (X_row, X_col)), dtype=int)
print "Finished transform to a sparse matrix " + str(time.now())

这里构建的是COO格式的矩阵，很容易转换成你喜欢的任何格式：

X = X.tocsr()

Python - 将稀疏文件读入稀疏矩阵的最佳方式

Python - The best way to read a sparse file into a sparse matrix

numpy

matrix

scipy

sparse-matrix