Bash 脚本：从文件中读取 30000 条记录并运行多个并行处理

Question

txt 超过 30000 条记录。所有记录都是一条一条的，是这样的一个IP：

192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5
192.168.0.6
192.168.0.7
192.168.0.8
192.168.0.9
192.168.0.10

我阅读了 bash 脚本中的每一行，我需要运行像这样的卷曲：

while IFS= read -r line || [[ -n "$line" ]]; do
    #check_site "$line"
    resp=$(curl -i -m1 http://$line 2>&1)
    echo "$resp" | grep -Eo "$ok" > /dev/null

    if [ $? -ne 0 ]; then
        #echo -e "failed: $line" >> "${logfile}"
        echo -e "Command: curl -i -m1 http://$line 2>&1" >> "${outfile}"
        echo -e "failed: $line:\n\n \"$resp\"\n\n" >> "${outfile}"
        echo "$line" >> "${faillog}"
    fi
done < "${FILE}"

有没有办法在我的文件中同时运行多行以减少执行时间？

Answer 1

本文介绍了解决并行执行的方法，可能对您有所帮助： Parallel execution in Bash

文章中的示例：

#!/bin/bash

RANDOM=10
JOBS_COUNTER=0
MAX_CHILDREN=10
MY_PID=$$

for i in {1..100}
do
    echo Cycle counter: $i
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    while [ $JOBS_COUNTER -ge $MAX_CHILDREN ]
    do
        JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
        echo Jobs counter: $JOBS_COUNTER
        sleep 1
    done
    sleep $(($RANDOM % 30)) &
done
echo Finishing children ...
# wait for children here
while [ $JOBS_COUNTER -gt 1 ]
do
    JOBS_COUNTER=$((`ps ax -Ao ppid | grep $MY_PID | wc -l`))
    echo Jobs counter: $JOBS_COUNTER
    sleep 1
done
echo Done

Answer 2

我是这样解决多进程的：

#export variable to be used into function    
export outlog="/tmp/out.log"
export faillog="/tmp/fail.log"
export ok="(curl: \(7\) Failed to connect to)" # acceptable responses

# create function:
check_site() {
  ip=
  resp=$(curl -i -m1 http://$ip 2>&1)
  echo "$resp" | grep -Eo "$ok" > /dev/null

  if [ $? -ne 0 ]; then
    echo -e "Command: curl -i -m1 http://$ip 2>&1" >> "${outlog}"
    echo -e "Block failed: $ip:\n\n \"$resp\"\n\n" >> "${outlog}"
    echo "$ip" >> "${faillog}"
  fi
}

# call the function:
export -f check_site
parallel -j 252 -a "${FILE}" check_site

Answer 3

Xargs 可以解决问题。 Wikipedia

Bash 脚本：从文件中读取 30000 条记录并运行多个并行处理

Bash script: read 30000 records from file and run multiple process in parallel

linux

bash

do-while

txt

script

Bash 脚本：从文件中读取 30000 条记录并 运行 多个并行处理

Bash script: read 30000 records from file and run multiple process in parallel

linux

bash

do-while

txt

script

Bash 脚本：从文件中读取 30000 条记录并运行多个并行处理