如何计算数据框的协方差矩阵

Question

我已经使用pandas read_fwf函数读取了传感器数据的数据帧。我需要找到读取 928991 x 8 矩阵的协方差矩阵。最终，我想找到特征向量和特征值，使用这个协方差矩阵的主成分分析算法。

Answer 1

首先，您需要使用 df.values 将 pandas 数据帧放入一个 numpy 数组。例如：

A = df.values

将数据放入 numpy 数组后，计算协方差矩阵或 PCA 会容易得多。更多：

# import functions you need to compute covariance matrix from numpy
from numpy import array
from numpy import mean
from numpy import cov
from numpy.linalg import eig

# assume you load your data using pd.read_fwf to variable *df*
df = pd.read_fwf(filepath, widths=col_widths, names=col_names)
#put dataframe values to a numpy array
A = df.values
#check matrix A's shape, it should be (928991, 8)
print(A.shape)
# calculate the mean of each column
M = mean(A.T, axis=1)
print(M)
# center columns by subtracting column means
C = A - M
print(C)
# calculate covariance matrix of centered matrix
V = cov(C.T)
print(V)
# eigendecomposition of covariance matrix
values, vectors = eig(V)
print(vectors)
print(values)
# project data
P = vectors.T.dot(C.T)
print(P.T)

运行该示例首先打印原始矩阵，然后是居中协方差矩阵的特征向量和特征值，最后是原始矩阵的投影。这是一个 link，您可能会发现对您的 PCA task 有用。

Answer 2

为什么不直接使用 pd.DataFrame.cov function？

Answer 3

本题答案如下

import pandas as pd
import numpy as np
from numpy.linalg import eig

df_sensor_data = pd.read_csv('HT_Sensor_dataset.dat', delim_whitespace=True)
del df_sensor_data['id']
del df_sensor_data['time']
del df_sensor_data['Temp.']
del df_sensor_data['Humidity']
df = df_sensor_data.notna().astype('float64')
covariance_matrix = df_sensor_data.cov()
print(covariance_matrix)

values, vectors = eig(covariance_matrix)
print(values)
print(vectors)

如何计算数据框的协方差矩阵

How to calculate covariance matrix of data frame

python

covariance

eigenvalue

pca