如何在不在 bash 脚本中创建竞争条件的情况下检查文件是否存在？
How to check if a file exists without creating a race condition in a bash script?
如果我错了，请纠正我，但我所知道的以及我对竞争条件和 TOCTOU（检查时间和使用时间）错误的理解，以这种方式检查文件是否存在：
if [ -f /path/to/file ]; then
#File exists do some operations on it
fi
创建竞争条件和 TOCTOU 错误。
那么有没有其他方法可以在不创建竞争条件的情况下检查文件或目录是否存在，或者如果它不存在则尝试打开文件并处理错误。
我知道在大多数脚本中使用以前的方法可能不是那么重要，但对我来说最好练习避免这种情况。
感谢您的帮助。
为避免竞争条件，您可以将文件重命名为第一步锁定条件。在许多文件系统上，这是一个“atomic”操作，无法同时完成（一个 inode 写入）。
这样，如果重命名成功，您可以确定该文件存在，并且您的 none 个其他进程已使用其原始名称。
例如，用当前进程 PID 重命名文件：
mv /path/to/file path/to/file.$$
if [ $? = 0 ] ; then
# Success, we can work on path/to/file.$$, and we're then the only one to do so from
# our processes point of view.
cat path/to/file.$$ # doing something with the file
# At the end, we can rename/move the file as 'processed'
mv path/to/file.$$ processed_path/to/file
fi
这样，您也可以对以PID号为扩展名的文件进行恢复。
编辑： 正如@Thomas 所主张的，这里是这个解决方案的基本实现，作为 bash 脚本，process。它被排除在像这样的目录树中：
[ `process` current directory ]
|-->[input] input directory where the script look for '*.txt' files to process
|-->[input_path_etl] input directory where the script will place processed file for ETL
该脚本需要 /proc 文件系统来进行简单的进程检查。
对于垂直可读性，SC2181 尚未应用。
该脚本使用 ./process 处理文件，并且可以使用 ./process -r 从其当前路径 进行崩溃恢复。这只是一个例子来说明如何使用 mv lock。此处处理 .txt 文件是将文件中的数据虚拟加载到数据库中的第一步，第二步是为 ETL 处理器生成虚拟文件。
#!/bin/bash
# process factory paths, should read from a config file, LDAP source, wathever...
process_input_path="input"
process_input_etl_path="input_path_etl"
# Example of an imaginary process that load stdin into a db
load_into_db() {
return 0;
}
# Example of an imaginary process that clean the data in the db for recovery
# Parameters : { filename }
# Returns: 0 successful recover, 1 otherwise
# filename: file path and name which require cleaning in db, mandatory
# stderr: potential cleaning erros
clean_db() {
if [ $# != 1 ] ; then
echo "ERROR: clean_db, wrong parameters" 2>&1
return 1;
fi
return 0;
}
# Example of an imaginary process that load a file into a db
# Parameters: { filename }
# returns: 0 if successfull, 1 if failed
# filename: file's path and name of the file to process, mandatory
# stderr: potential processing errors
process_first_step() {
if [ $# != 1 ] ; then
echo "ERROR: process_first_step wrong parameters" 2>&1
return 1;
fi
# first example step, load things from the file into a db
cat ".$$_1" | load_into_db
if [ $? = 0 ] ; then
# rename first the file to means step 1 was succesfully done and
# we go for the second
mv ".$$_1" ".$$_2"
if [ $? = 0 ] ; then
# success, the file is ready for step 2
return 0;
fi
fi
# If we're here, something went wrong in step 1, exiting with error
return 1;
}
# Example of an imaginary process that put a file into the input path of an ETL
# Parameters: { filename }
# returns: 0 if successfull, 1 if failed
# filename: file's path and name of the file to process, mandatory
# stderr: potential processing errors
process_second_step() {
if [ $# != 1 ] ; then
echo "ERROR: process_second_step wrong parameters" 2>&1
return 1;
fi
# the file is ready for step 2, we create the appropriate input
# for the ETL with some sed transfomration beforehand
cat ".$$_2" | sed 's/line/lInE/g' > ".$$_2.etl"
if [ $? = 0 ] ; then
# Success, the file is ready for the ETL factory process,
# we move it in with an atomic mv to make it visible from
# the ETL factory process
mv ".$$_2.etl" "${process_input_etl_path}/"
if [ $? = 0 ] ; then
# Successful, step 2 is done
return 0;
fi
fi
# If we're here, something went wrong in step 2, exiting with error
return 1;
}
# Example of an imaginary file processor that conducts all the
# required step on the provided file
# Parameters : { filename }
# Returns : 0 if successful, 1 otherwise
# filename : file's path and name of the file to process, mandatory
# stderr: potential processing errors
process_file() {
if [ $# != 1 ] ; then
echo "ERROR: process_file, wrong parameters" 2>&1
return 1;
fi
# Lock the file for processing step one
mv "" ".$$_1"
if [ $? = 0 ] ; then
# ok we have the file for us
# first example step, load things from the file into a db
process_first_step ""
if [ $? = 0 ] ; then
# first step is successful, so continue the process
# next example step, add the loaded lines with transformations into the input path of antoher process factory (like an ETL)
process_second_step ""
if [ $? = 0 ] ; then
# Second step is susccesful, we can now rename the file with
# a suffix meaning it was fully processed, a filename that would
# not be visible for the factory process
mv ".$$_2" "_processed"
if [ $? != 0 ] ; then
# if this failed, we have to return an error,
# the current file name would be .$$_2, not visible
# from the process factory and the error message will mean
# that the file was fully processed but can't be renamed
# at the end, so no recovering is required
echo "ERROR: process_file, can't be renamed as fully processed." 2>&1
return 1;
fi
# if we're here, the file was fully processed and rename accordingly,
# we return a success status
return 0;
fi
fi
fi
# If we're here, something went wrong in the process, we exit with an error
# the actual filename will be .$$_1 or .$$_2 depending of where it was
# in the processing chain, it will not be visible from the main
# process factory and the rcovery process can then process it accordingly
return 1;
}
# Example of an imaginary process recovery for orphan files due to a crash,
# power outage, unexpected reboot, CTRL^C, etc.
# Returns: 0 for success, 1 if error(s)
# stdout: recovery operations infos if any
# stderr: potential error(s)
process_recovery() {
if [ $# != 0 ] ; then
echo "ERROR: process_recovery, wrong parameters." 2>&1
return 1;
fi
# local variables
local process_FILE=""
local process_PID=""
local process_STEP=""
local process_CMD=""
# flag for the file that means :
# 0 : do not recover
# 1 : recover
# 2 : can't recover
# 3 : recover successful, rename to put the file back in the process
# 4 : recover successful, rename it as fully processed
local recover_status=0
# flag to check if the recover process is succesful,
# 0: success
# 1: error(s)
local recovery_status=0
# We can only have one recovery process at a time, check for the corresponding lock, we use an atomic mkdir for that
mkdir "${process_input_path}/recover" &>/dev/null
if [ $? != 0 ] ; then
# if it fails, it means there is probably already a running recover
echo "ERROR: process_recovery, a recovery seems to be still in progress." 2>&1
echo " if there is no more running recovery (crash)," 2>&1
echo " disarm manually the lock by removing the recover folder." 2>&1
echo " Check also that the input folder is writable for script." 2>&1
return 1;
fi
# We first have to check every files in the input path that match
# a *.txt.<PID>_<step> pattern
find "${process_input_path}/" -name '*.txt.[0-9]*_[12]' | ( while read -r file_to_check || exit ${recovery_status}; do
# By default, do not recover
recover_status=0
# Get the PID and check if there is a running corresponding process
process_PID="$(echo "${file_to_check}" | sed 's/^.*\.txt\.\([^_]*\)_[0-9]*$//')"
if [[ $? != 0 || "${process_PID}" = "${file_to_check}" ]] ; then
# Something went wrong, we output an error on stderr and set the flag
echo "ERROR: process_recovery, failed to parse pid from file name ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
# We check the shell process through /proc and check it is our
process_CMD="$(cat "/proc/${process_PID}/comm" 2>/dev/null)"
if [[ $? = 0 && "$(echo "${process_CMD}" | grep process.sh)" != "" ]] ; then
# There is a process.sh with the same PID, no recover needed
echo "File ${file_to_check} is processed by PID ${process_PID}..."
else
# There is no corresponding process, but it could have finished during
# our operations, so we check if the file is still here
if [ -e "${file_to_check}" ] ; then
# The file is still here, so we need to recover
echo "XX${process_CMD}"
recover_status=1;
fi
fi
fi
if [ "${recover_status}" = "1" ] ; then
# The file should be recovered, signal it
echo "Recovering file ${file_to_check}..."
# Get the original file name
process_FILE="$(echo "${file_to_check}" | sed 's/^\(.*\.txt\)\.[^_]*_[0-9]*$//')"
if [[ $? != 0 || "${process_FILE}" = "${file_to_check}" ]] ; then
# Something went wrong, we output an error on stderr and set the flag
echo "ERROR: process_recovery, failed to parse original name from file name ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
# We need to know at which step it was
process_STEP="$(echo "${file_to_check}" | sed 's/^.*\.txt\.[^_]*_\([0-9]*\)$//')"
if [[ $? != 0 || "${process_STEP}" = "${file_to_check}" ]] ; then
# Something went wrong, we output an error on stderr and set the flag
echo "ERROR: process_recovery, failed to parse step from file name ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
fi
fi
# Still ok to recover ?
if [ "${recover_status}" = "1" ] ; then
# check the step
case "${process_STEP}" in
"1")
# Do database cleaning for the file, we will revert and rename the file
# so it will be processed next by the factory process
clean_db "${file_to_check}"
if [ $? != 0 ]; then
# The cleaning process has failed, signal it
echo "ERROR: process_recovery, failed to clean the db for ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
# Cleaning was successful, rename the file so it will be
# visible at new from the process factory
recover_status=3;
fi
;;
"2")
# If the file is still here, check if it is not in the input path of the ETL
# or if the ETL is/has already processing/processed it
if [[ -e "${process_input_etl_path}/${process_FILE}.etl" || -e "${process_input_etl_path}/${process_FILE}.etl_processed" ]] ; then
# The file as fully completed step 2 then and should be marked as processed
recover_status=4;
else
# If the file has not reach the ETL input path, we just have to launch step 2 for the file
# If there is .etl local file, we aren't sure it was completed before crash, so a redo of step will simply overwrite it,
# as it is a local file in the current path, it has never been seen by the ETL
# We rename it for processing with the recovery PID
echo "Recovering ${file_to_check} on step 2 as ${process_FILE}.$$_2..."
mv "${file_to_check}" "${process_FILE}.$$_2"
if [ $? != 0 ]; then
# The renaming failed, signal it. The file is still here, so a future recovery can handle it , nothing more to do
echo "ERROR: process_recovery, failed to rename file ${file_to_check} for step 2" 2>&1
recovery_status=1;
recover_status=2;
else
# File is ready for step 2
process_second_step "${process_FILE}"
if [ $? != 0 ]; then
# The step 2 redo failed, signal it. The file is still here, so a future recovery can handle it , nothing more to do
echo "ERROR: process_recovery, failed to redo step 2 for ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
# The file as fully completed step 2 then and should be marked as processed
recover_status=4;
# Need so that the processed part deals with the new filename
file_to_check="${process_FILE}.$$_2"
fi
fi
fi
;;
*)
# Abnormal situation, unknow step, signal it
echo "ERROR: process_recovery, unknown step for ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
;;
esac;
# If the recovery operations were successful, we can now rename the file accordingly
case "${recover_status}" in
"3")
# Rename it 'back' so the file will be processed by the process factory next
mv "${file_to_check}" "${process_FILE}"
if [ $? != 0 ]; then
# The renaming failed, signal it. The file is still here, so a future recovery can handle it , nothing more to do
echo "ERROR: process_recovery, failed to put back the file ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
echo "Recovering ${file_to_check}...done, reverted."
fi
;;
"4")
# Rename as already fully processed
mv "${file_to_check}" "${process_FILE}_processed"
if [ $? != 0 ]; then
# The renaming failed, signal it. The file is still here, so a future recovery can handle it , nothing more to do
echo "ERROR: process_recovery, failed to rename the fully processed file ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
echo "Recovering ${file_to_check}...done, processed."
fi
;;
esac;
fi
fi
done )
if [ $? != 0 ] ; then
# the recovery processing meets errors, we have to exit with error
recovery_status=1;
fi
# Finished, we can remove the recovery lock, there'll b nop race condition if a second recovery process start now
# We can only have one recovery process at a time, check for the corresponding lock, we use an atomic mkdir for that
rmdir "${process_input_path}/recover" &>/dev/null
if [ $? != 0 ] ; then
echo "ERROR: process_recovery, can't remove the recovery lock, you'l have to manually remove it." 2>&1
recovery_status=1;
fi
# Return status
return ${recovery_status};
}
# Example of an imaginary file processing factory
# this factory will look for all files matching '*.txt' in its input path
# Parameteres: [ -r ]
# Returns : 0 if all matching files in the input path were processed,
# 1 otherwise
# -r : Instead of processing files, launch the recovery process, optional
# stdout : processing log
# stderr : potential processing errors
process_files() {
if [ $# -gt 1 ]; then
echo "ERROR: process_files, wrong parameters" 2>&1
return 1;
fi
if [[ $# = 1 && "" = "-r" ]] ; then
# launch the recovery process and exit its exit status
process_recovery
return $?
fi
if [ $# != 0 ] ; then
echo "ERROR: process_files, unknown parametrs : $*" 2>&1
return 1;
fi
# Parameter(s) have been processed, we are now looking for files to process
local process_status=0;
find "${process_input_path}/" -name '*.txt' | ( while read -r file_to_process || exit ${process_status}; do
echo "Processing ${file_to_process}..."
process_file "${file_to_process}"
if [ $? != 0 ] ; then
# Something went wrong, signal it on stderr
echo "Processing ${file_to_process} failed, the file may has been locked by antoher process or may be in the wrong format." 2>&1
# We set the flag for signaling trouble but we continue to process
# the following files
process_status=1;
else
echo "Processing ${file_to_process}...done."
fi
done )
if [ $? != 0 ] ; then
# the factory processing meets errors, we have to exit with error
return 1;
fi
# All matching files were correctly processed or there was no
# matching files to process, we return a success
return 0;
}
# The main entry point
# check that we have paths before anything harmful happend..
if [[ -z "${process_input_path}" || -z "${process_input_etl_path}" ]] ; then
echo "ERROR: [=11=], configuration missing..." 2>&1
exit 1;
fi
# Before processing any file, we check for /proc
if [ ! -e "/proc/$$" ] ; then
echo "ERROR: [=11=], /proc is required..." 2>&1
exit 2;
fi
# We force a common identifier for the processing script, process.sh, so recovery can easily check for running process
echo "process.sh" > "/proc/$$/comm"
if [ $? != 0 ] ; then
echo "ERROR: [=11=], can't set /proc/$$/comm..." 2>&1
exit 3;
fi
process_files "$@"
您的代码编写方式，if 检查某个文件是否存在，如果存在则执行 if 的主体。
现在，如果 if 的主体依赖于文件存在这一事实（由条件确定），那么实际上只有竞争条件。当正文对文件执行某些操作时，通常会出现这种情况，例如：
- 打开文件进行读取/追加/写入（如：仅写入存在的文件）
- 移动或复制文件
- 正在查询文件的一些属性（大小、最后修改时间等）
然而，其中许多操作可以在不首先检查文件是否存在的情况下执行。相反，您可以简单地执行操作并对“找不到文件”错误做出反应。
然而，如果您想执行多个这样的操作并排除相关文件被同时更改的可能性，这就变得更加棘手了。这不能通过简单的 if 来完成，因为语义略有不同：
- 在你的问题中写了一个
if，你正在检查：文件在这个时间点是否存在？（时间点是执行条件)
- 要成功 运行 没有竞态条件的整个主体，您需要做的是检查 文件是否在主体的整个持续时间内存在（并保持不变）？
你看前者不能保证后者。您需要某种锁定机制。
如果我错了，请纠正我，但我所知道的以及我对竞争条件和 TOCTOU（检查时间和使用时间）错误的理解，以这种方式检查文件是否存在：
if [ -f /path/to/file ]; then
#File exists do some operations on it
fi
创建竞争条件和 TOCTOU 错误。 那么有没有其他方法可以在不创建竞争条件的情况下检查文件或目录是否存在，或者如果它不存在则尝试打开文件并处理错误。
我知道在大多数脚本中使用以前的方法可能不是那么重要，但对我来说最好练习避免这种情况。
感谢您的帮助。
为避免竞争条件，您可以将文件重命名为第一步锁定条件。在许多文件系统上，这是一个“atomic”操作，无法同时完成（一个 inode 写入）。
这样，如果重命名成功，您可以确定该文件存在，并且您的 none 个其他进程已使用其原始名称。
例如，用当前进程 PID 重命名文件：
mv /path/to/file path/to/file.$$
if [ $? = 0 ] ; then
# Success, we can work on path/to/file.$$, and we're then the only one to do so from
# our processes point of view.
cat path/to/file.$$ # doing something with the file
# At the end, we can rename/move the file as 'processed'
mv path/to/file.$$ processed_path/to/file
fi
这样，您也可以对以PID号为扩展名的文件进行恢复。
编辑： 正如@Thomas 所主张的，这里是这个解决方案的基本实现，作为 bash 脚本，process。它被排除在像这样的目录树中：
[ `process` current directory ] |-->[input] input directory where the script look for '*.txt' files to process |-->[input_path_etl] input directory where the script will place processed file for ETL
该脚本需要 /proc 文件系统来进行简单的进程检查。
对于垂直可读性，SC2181 尚未应用。
该脚本使用 ./process 处理文件，并且可以使用 ./process -r 从其当前路径 进行崩溃恢复。这只是一个例子来说明如何使用 mv lock。此处处理 .txt 文件是将文件中的数据虚拟加载到数据库中的第一步，第二步是为 ETL 处理器生成虚拟文件。
#!/bin/bash
# process factory paths, should read from a config file, LDAP source, wathever...
process_input_path="input"
process_input_etl_path="input_path_etl"
# Example of an imaginary process that load stdin into a db
load_into_db() {
return 0;
}
# Example of an imaginary process that clean the data in the db for recovery
# Parameters : { filename }
# Returns: 0 successful recover, 1 otherwise
# filename: file path and name which require cleaning in db, mandatory
# stderr: potential cleaning erros
clean_db() {
if [ $# != 1 ] ; then
echo "ERROR: clean_db, wrong parameters" 2>&1
return 1;
fi
return 0;
}
# Example of an imaginary process that load a file into a db
# Parameters: { filename }
# returns: 0 if successfull, 1 if failed
# filename: file's path and name of the file to process, mandatory
# stderr: potential processing errors
process_first_step() {
if [ $# != 1 ] ; then
echo "ERROR: process_first_step wrong parameters" 2>&1
return 1;
fi
# first example step, load things from the file into a db
cat ".$$_1" | load_into_db
if [ $? = 0 ] ; then
# rename first the file to means step 1 was succesfully done and
# we go for the second
mv ".$$_1" ".$$_2"
if [ $? = 0 ] ; then
# success, the file is ready for step 2
return 0;
fi
fi
# If we're here, something went wrong in step 1, exiting with error
return 1;
}
# Example of an imaginary process that put a file into the input path of an ETL
# Parameters: { filename }
# returns: 0 if successfull, 1 if failed
# filename: file's path and name of the file to process, mandatory
# stderr: potential processing errors
process_second_step() {
if [ $# != 1 ] ; then
echo "ERROR: process_second_step wrong parameters" 2>&1
return 1;
fi
# the file is ready for step 2, we create the appropriate input
# for the ETL with some sed transfomration beforehand
cat ".$$_2" | sed 's/line/lInE/g' > ".$$_2.etl"
if [ $? = 0 ] ; then
# Success, the file is ready for the ETL factory process,
# we move it in with an atomic mv to make it visible from
# the ETL factory process
mv ".$$_2.etl" "${process_input_etl_path}/"
if [ $? = 0 ] ; then
# Successful, step 2 is done
return 0;
fi
fi
# If we're here, something went wrong in step 2, exiting with error
return 1;
}
# Example of an imaginary file processor that conducts all the
# required step on the provided file
# Parameters : { filename }
# Returns : 0 if successful, 1 otherwise
# filename : file's path and name of the file to process, mandatory
# stderr: potential processing errors
process_file() {
if [ $# != 1 ] ; then
echo "ERROR: process_file, wrong parameters" 2>&1
return 1;
fi
# Lock the file for processing step one
mv "" ".$$_1"
if [ $? = 0 ] ; then
# ok we have the file for us
# first example step, load things from the file into a db
process_first_step ""
if [ $? = 0 ] ; then
# first step is successful, so continue the process
# next example step, add the loaded lines with transformations into the input path of antoher process factory (like an ETL)
process_second_step ""
if [ $? = 0 ] ; then
# Second step is susccesful, we can now rename the file with
# a suffix meaning it was fully processed, a filename that would
# not be visible for the factory process
mv ".$$_2" "_processed"
if [ $? != 0 ] ; then
# if this failed, we have to return an error,
# the current file name would be .$$_2, not visible
# from the process factory and the error message will mean
# that the file was fully processed but can't be renamed
# at the end, so no recovering is required
echo "ERROR: process_file, can't be renamed as fully processed." 2>&1
return 1;
fi
# if we're here, the file was fully processed and rename accordingly,
# we return a success status
return 0;
fi
fi
fi
# If we're here, something went wrong in the process, we exit with an error
# the actual filename will be .$$_1 or .$$_2 depending of where it was
# in the processing chain, it will not be visible from the main
# process factory and the rcovery process can then process it accordingly
return 1;
}
# Example of an imaginary process recovery for orphan files due to a crash,
# power outage, unexpected reboot, CTRL^C, etc.
# Returns: 0 for success, 1 if error(s)
# stdout: recovery operations infos if any
# stderr: potential error(s)
process_recovery() {
if [ $# != 0 ] ; then
echo "ERROR: process_recovery, wrong parameters." 2>&1
return 1;
fi
# local variables
local process_FILE=""
local process_PID=""
local process_STEP=""
local process_CMD=""
# flag for the file that means :
# 0 : do not recover
# 1 : recover
# 2 : can't recover
# 3 : recover successful, rename to put the file back in the process
# 4 : recover successful, rename it as fully processed
local recover_status=0
# flag to check if the recover process is succesful,
# 0: success
# 1: error(s)
local recovery_status=0
# We can only have one recovery process at a time, check for the corresponding lock, we use an atomic mkdir for that
mkdir "${process_input_path}/recover" &>/dev/null
if [ $? != 0 ] ; then
# if it fails, it means there is probably already a running recover
echo "ERROR: process_recovery, a recovery seems to be still in progress." 2>&1
echo " if there is no more running recovery (crash)," 2>&1
echo " disarm manually the lock by removing the recover folder." 2>&1
echo " Check also that the input folder is writable for script." 2>&1
return 1;
fi
# We first have to check every files in the input path that match
# a *.txt.<PID>_<step> pattern
find "${process_input_path}/" -name '*.txt.[0-9]*_[12]' | ( while read -r file_to_check || exit ${recovery_status}; do
# By default, do not recover
recover_status=0
# Get the PID and check if there is a running corresponding process
process_PID="$(echo "${file_to_check}" | sed 's/^.*\.txt\.\([^_]*\)_[0-9]*$//')"
if [[ $? != 0 || "${process_PID}" = "${file_to_check}" ]] ; then
# Something went wrong, we output an error on stderr and set the flag
echo "ERROR: process_recovery, failed to parse pid from file name ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
# We check the shell process through /proc and check it is our
process_CMD="$(cat "/proc/${process_PID}/comm" 2>/dev/null)"
if [[ $? = 0 && "$(echo "${process_CMD}" | grep process.sh)" != "" ]] ; then
# There is a process.sh with the same PID, no recover needed
echo "File ${file_to_check} is processed by PID ${process_PID}..."
else
# There is no corresponding process, but it could have finished during
# our operations, so we check if the file is still here
if [ -e "${file_to_check}" ] ; then
# The file is still here, so we need to recover
echo "XX${process_CMD}"
recover_status=1;
fi
fi
fi
if [ "${recover_status}" = "1" ] ; then
# The file should be recovered, signal it
echo "Recovering file ${file_to_check}..."
# Get the original file name
process_FILE="$(echo "${file_to_check}" | sed 's/^\(.*\.txt\)\.[^_]*_[0-9]*$//')"
if [[ $? != 0 || "${process_FILE}" = "${file_to_check}" ]] ; then
# Something went wrong, we output an error on stderr and set the flag
echo "ERROR: process_recovery, failed to parse original name from file name ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
# We need to know at which step it was
process_STEP="$(echo "${file_to_check}" | sed 's/^.*\.txt\.[^_]*_\([0-9]*\)$//')"
if [[ $? != 0 || "${process_STEP}" = "${file_to_check}" ]] ; then
# Something went wrong, we output an error on stderr and set the flag
echo "ERROR: process_recovery, failed to parse step from file name ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
fi
fi
# Still ok to recover ?
if [ "${recover_status}" = "1" ] ; then
# check the step
case "${process_STEP}" in
"1")
# Do database cleaning for the file, we will revert and rename the file
# so it will be processed next by the factory process
clean_db "${file_to_check}"
if [ $? != 0 ]; then
# The cleaning process has failed, signal it
echo "ERROR: process_recovery, failed to clean the db for ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
# Cleaning was successful, rename the file so it will be
# visible at new from the process factory
recover_status=3;
fi
;;
"2")
# If the file is still here, check if it is not in the input path of the ETL
# or if the ETL is/has already processing/processed it
if [[ -e "${process_input_etl_path}/${process_FILE}.etl" || -e "${process_input_etl_path}/${process_FILE}.etl_processed" ]] ; then
# The file as fully completed step 2 then and should be marked as processed
recover_status=4;
else
# If the file has not reach the ETL input path, we just have to launch step 2 for the file
# If there is .etl local file, we aren't sure it was completed before crash, so a redo of step will simply overwrite it,
# as it is a local file in the current path, it has never been seen by the ETL
# We rename it for processing with the recovery PID
echo "Recovering ${file_to_check} on step 2 as ${process_FILE}.$$_2..."
mv "${file_to_check}" "${process_FILE}.$$_2"
if [ $? != 0 ]; then
# The renaming failed, signal it. The file is still here, so a future recovery can handle it , nothing more to do
echo "ERROR: process_recovery, failed to rename file ${file_to_check} for step 2" 2>&1
recovery_status=1;
recover_status=2;
else
# File is ready for step 2
process_second_step "${process_FILE}"
if [ $? != 0 ]; then
# The step 2 redo failed, signal it. The file is still here, so a future recovery can handle it , nothing more to do
echo "ERROR: process_recovery, failed to redo step 2 for ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
# The file as fully completed step 2 then and should be marked as processed
recover_status=4;
# Need so that the processed part deals with the new filename
file_to_check="${process_FILE}.$$_2"
fi
fi
fi
;;
*)
# Abnormal situation, unknow step, signal it
echo "ERROR: process_recovery, unknown step for ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
;;
esac;
# If the recovery operations were successful, we can now rename the file accordingly
case "${recover_status}" in
"3")
# Rename it 'back' so the file will be processed by the process factory next
mv "${file_to_check}" "${process_FILE}"
if [ $? != 0 ]; then
# The renaming failed, signal it. The file is still here, so a future recovery can handle it , nothing more to do
echo "ERROR: process_recovery, failed to put back the file ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
echo "Recovering ${file_to_check}...done, reverted."
fi
;;
"4")
# Rename as already fully processed
mv "${file_to_check}" "${process_FILE}_processed"
if [ $? != 0 ]; then
# The renaming failed, signal it. The file is still here, so a future recovery can handle it , nothing more to do
echo "ERROR: process_recovery, failed to rename the fully processed file ${file_to_check}" 2>&1
recovery_status=1;
recover_status=2;
else
echo "Recovering ${file_to_check}...done, processed."
fi
;;
esac;
fi
fi
done )
if [ $? != 0 ] ; then
# the recovery processing meets errors, we have to exit with error
recovery_status=1;
fi
# Finished, we can remove the recovery lock, there'll b nop race condition if a second recovery process start now
# We can only have one recovery process at a time, check for the corresponding lock, we use an atomic mkdir for that
rmdir "${process_input_path}/recover" &>/dev/null
if [ $? != 0 ] ; then
echo "ERROR: process_recovery, can't remove the recovery lock, you'l have to manually remove it." 2>&1
recovery_status=1;
fi
# Return status
return ${recovery_status};
}
# Example of an imaginary file processing factory
# this factory will look for all files matching '*.txt' in its input path
# Parameteres: [ -r ]
# Returns : 0 if all matching files in the input path were processed,
# 1 otherwise
# -r : Instead of processing files, launch the recovery process, optional
# stdout : processing log
# stderr : potential processing errors
process_files() {
if [ $# -gt 1 ]; then
echo "ERROR: process_files, wrong parameters" 2>&1
return 1;
fi
if [[ $# = 1 && "" = "-r" ]] ; then
# launch the recovery process and exit its exit status
process_recovery
return $?
fi
if [ $# != 0 ] ; then
echo "ERROR: process_files, unknown parametrs : $*" 2>&1
return 1;
fi
# Parameter(s) have been processed, we are now looking for files to process
local process_status=0;
find "${process_input_path}/" -name '*.txt' | ( while read -r file_to_process || exit ${process_status}; do
echo "Processing ${file_to_process}..."
process_file "${file_to_process}"
if [ $? != 0 ] ; then
# Something went wrong, signal it on stderr
echo "Processing ${file_to_process} failed, the file may has been locked by antoher process or may be in the wrong format." 2>&1
# We set the flag for signaling trouble but we continue to process
# the following files
process_status=1;
else
echo "Processing ${file_to_process}...done."
fi
done )
if [ $? != 0 ] ; then
# the factory processing meets errors, we have to exit with error
return 1;
fi
# All matching files were correctly processed or there was no
# matching files to process, we return a success
return 0;
}
# The main entry point
# check that we have paths before anything harmful happend..
if [[ -z "${process_input_path}" || -z "${process_input_etl_path}" ]] ; then
echo "ERROR: [=11=], configuration missing..." 2>&1
exit 1;
fi
# Before processing any file, we check for /proc
if [ ! -e "/proc/$$" ] ; then
echo "ERROR: [=11=], /proc is required..." 2>&1
exit 2;
fi
# We force a common identifier for the processing script, process.sh, so recovery can easily check for running process
echo "process.sh" > "/proc/$$/comm"
if [ $? != 0 ] ; then
echo "ERROR: [=11=], can't set /proc/$$/comm..." 2>&1
exit 3;
fi
process_files "$@"
您的代码编写方式，if 检查某个文件是否存在，如果存在则执行 if 的主体。
现在，如果 if 的主体依赖于文件存在这一事实（由条件确定），那么实际上只有竞争条件。当正文对文件执行某些操作时，通常会出现这种情况，例如：
- 打开文件进行读取/追加/写入（如：仅写入存在的文件）
- 移动或复制文件
- 正在查询文件的一些属性（大小、最后修改时间等）
然而，其中许多操作可以在不首先检查文件是否存在的情况下执行。相反，您可以简单地执行操作并对“找不到文件”错误做出反应。
然而，如果您想执行多个这样的操作并排除相关文件被同时更改的可能性，这就变得更加棘手了。这不能通过简单的 if 来完成，因为语义略有不同：
- 在你的问题中写了一个
if，你正在检查：文件在这个时间点是否存在？（时间点是执行条件) - 要成功 运行 没有竞态条件的整个主体，您需要做的是检查 文件是否在主体的整个持续时间内存在（并保持不变）？
你看前者不能保证后者。您需要某种锁定机制。