如何确保数据框已通过 pandas.to_csv() 完成写入？

Question

我一直在创建一个查询数据库和 returns 结果的小脚本。然后，在将该 CSV 结果上传到云位置之前，我一直在使用 Pandas.to_csv() 将其写入 CSV 临时文件。我运行遇到的麻烦是确保 pandas.to_csv() 函数在我将其上传到云位置之前完成写入 CSV 临时文件。我一直确保该日期在上传之前进入临时文件的唯一方法是保持

print(temp.tell())

下面示例中的代码行。如果我将其注释掉，则不会上传任何数据。

示例代码如下：

def write_to_temporary_csv_file(df, file_name, token, folder_id):
   with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as temp:
       print("DataFrame: ", df)
       df.to_csv(temp, index=False, encoding='utf-8')
       print("temp.tell() size: ", temp.tell())
       print("File size: ", str(round((os.stat(temp.name).st_size/1024), 2)), "kb")
       new_file_path = tempfile.gettempdir() + '/' + customer_name + '_' + file_name + '_' +  current_date + '.csv'

       ## Check if newly created renamed temp file already exist, if it does remove it to create it
       remove_temporary_file(new_file_path)
       os.link(temp.name, new_file_path)
       upload_response = upload_file(token, folder_id, new_file_path)

       ## Remove both the temp file and the newly created renamed temp file
       remove_temporary_file(temp.name)
       remove_temporary_file(new_file_path)

图片 1（包含 temp.tell()：图 2（temp.tell() 被注释掉：

Answer 1

我认为这可能是因为您一直打开文件（只要您在 with 块内）。这可能会导致内容未刷新到磁盘。

def write_to_temporary_csv_file(df, file_name, token, folder_id):
   with tempfile.NamedTemporaryFile(mode='w', suffix='.csv', delete=False) as temp:
       print("DataFrame: ", df)
       df.to_csv(temp, index=False, encoding='utf-8')

   # at this point we can close the file by exiting the with block

   print("temp.tell() size: ", temp.tell())
   print("File size: ", str(round((os.stat(temp.name).st_size/1024), 2)), "kb")
   new_file_path = tempfile.gettempdir() + '/' + customer_name + '_' + file_name + '_' +  current_date + '.csv'

   ## Check if newly created renamed temp file already exist, if it does remove it to create it
   remove_temporary_file(new_file_path)
   os.link(temp.name, new_file_path)
   upload_response = upload_file(token, folder_id, new_file_path)

   ## Remove both the temp file and the newly created renamed temp file
   remove_temporary_file(temp.name)
   remove_temporary_file(new_file_path)

如何确保数据框已通过 pandas.to_csv() 完成写入？

How can I ensure a dataframe has completed being written via pandas.to_csv()?

python

temporary-files

pandas