如何获取 PickleType 数据并使用 Python 将其转换为文件夹中的 csv 文件

Question

我正在尝试从 sqlalchemy 中获取 PickleType 数据并将其转换为文件夹中的 csv 文件。

这是我的 class 和 PickleType 数据的片段：

from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column
from sqlalchemy.sql.sqltypes import PickleType 
Base = declarative_base()

class Sales(Base):
    .....
    data = Column(PickleType, nullable=False, comment="Sale Data for sale")

这是我的函数，我试图在 PickleType 中获取数据并将其转换为文件夹中的 CSV 文件

from data import Sales
import pandas as pd
import os.path
import pickle

def _to_csv(self, sales_data : Sales):
    """
    To get Sales data to output CSV files into a folder
    """
    data1 = []
    with open(sales_data.data, 'rb') as file:
        data2 = pickle.dumps(file)
        data1 = pickle.load(data2)
        df = pd.DataFrame(data1)
        df.to_csv(os.path.join('/Desktop/sales','/car_sales.csv'))
        file.close()

Answer 1

我不知道你的 class 和数据的结构，但如果你有数据，你可以将它们附加到列表中并并行调用你的 _to_csv 方法来保存它为你。像这样的东西：


from data import Sales
import pandas as pd
import os.path
import pickle
import multiprocessing as mp
import concurrent.futures


def _to_csv(self, data):
    """
    To get Sales data to output CSV files into a folder
    """
    data1 = []
    with open(data 'rb') as file:
        data2 = pickle.dumps(file)
        data1 = pickle.load(data2)
        df = pd.DataFrame(data1)
        df.to_csv(os.path.join('/Desktop/sales',f'/{data}.csv'))
        file.close()

# I am assuming that you have the name of the files that you wanna read and save as cvs
data_list = []
for idx in range(20):
    name = f'car_sales{idx}'
    data_list.append(name)    

with concurrent.futures.ProcessPoolExecutor(max_workers=mp.cpu_count()) as executor:
    executor.map(_to_csv, data_list)

或者如果您想将每个数据名称分配给它的结果，您可以这样做：

with concurrent.futures.ProcessPoolExecutor(max_workers=mp.cpu_count()) as executor:
    futures ={executor.submit(_to_csv, data):data for data in data_list}
    for future in concurrent.futures.as_completed(futures):
        data = futures[future]
        try:
            result = future.result()
        except Exception as why:
            print('sth went wrong!', why)
        else:
            print(data, result)

如何获取 PickleType 数据并使用 Python 将其转换为文件夹中的 csv 文件

How to take a PickleType data and convert to csv files in a folder using Python

python

pandas

sqlalchemy

pickle