如何等待gcsfuse写通(flush)到GCS存储桶?
How to wait for gcsfuse to write-through (flush) to the GCS storage bucket?
在 Compute Engine 工作节点将文件写入 gcsfuse 安装的本地目录并关闭它们后,我希望它 将数据同步刷新到 GCS,然后通知其他工作节点所有文件已准备就绪。这是为了确保工作人员之间的同步。
Q.如何让gcsfuse直写到GCS,然后等待完成?
想法:
- 运行 Linux
sync
命令?
- 卸载目录然后等待
fusermount
命令到 return? (除了write-through时间,下一个worker任务unmount然后remount的时间会不会超过一两秒?)
- 让此任务中的所有程序在其所有输出文件上调用
fsync()
?那将是具有挑战性的。
- 多写一个文件,然后
flush()
和fsync()
那个?
Inodes may be opened for writing. Modifications are reflected
immediately in reads of the same inode by processes local to the
machine using the same file system. After a successful fsync or a
successful close, the contents of the inode are guaranteed to have
been written to the GCS object with the matching name if the object's
generation and meta-generation numbers still matched the source
generation of the inode. (They may not have if there had been
modifications from another actor in the meantime.) There are no
guarantees about whether local modifications are reflected in GCS
after writing but before syncing or closing.
因此,如果您的工作人员在写入文件后关闭文件,则后续依赖项应该能够始终如一地看到它们。
在 Compute Engine 工作节点将文件写入 gcsfuse 安装的本地目录并关闭它们后,我希望它 将数据同步刷新到 GCS,然后通知其他工作节点所有文件已准备就绪。这是为了确保工作人员之间的同步。
Q.如何让gcsfuse直写到GCS,然后等待完成?
想法:
- 运行 Linux
sync
命令? - 卸载目录然后等待
fusermount
命令到 return? (除了write-through时间,下一个worker任务unmount然后remount的时间会不会超过一两秒?) - 让此任务中的所有程序在其所有输出文件上调用
fsync()
?那将是具有挑战性的。 - 多写一个文件,然后
flush()
和fsync()
那个?
Inodes may be opened for writing. Modifications are reflected immediately in reads of the same inode by processes local to the machine using the same file system. After a successful fsync or a successful close, the contents of the inode are guaranteed to have been written to the GCS object with the matching name if the object's generation and meta-generation numbers still matched the source generation of the inode. (They may not have if there had been modifications from another actor in the meantime.) There are no guarantees about whether local modifications are reflected in GCS after writing but before syncing or closing.
因此,如果您的工作人员在写入文件后关闭文件,则后续依赖项应该能够始终如一地看到它们。