比较文件并在新文件中注释相同的行
Compare files and comment the same lines in new file
目标：我想比较两个 Suricata 规则文件并注释掉 file1 和 file2 中的相同行（警告“SID”），除非它已经被注释掉。我知道有更好的方法可以使用 Suricata 阈值文件来执行此操作，但不幸的是，我没有超出我可以在这里解释的那种奢侈。这是为了便于更新规则，其中规则可能会更新，但“SID”在两个文件中的共性将相同。
我不知道从哪里开始。
示例文件 1 文本：
alert $home_net any > $External_net any (msg: example; content: something; sid: 12345; rev:1)
#alert $home_net any > $External_net any (msg: example; content: something; sid: 67895; rev:1)
alert $home_net any > $External_net any (msg: example; content: something; sid: 18975; rev:1)
示例文件 2 文本：
alert $home_net any > $External_net any (msg: example; content: something; sid: 12345; rev:1)
<insert #>alert $home_net any > $External_net any (msg: example; content: something; sid: 67895; rev:1)
alert $home_net any > $External_net any (msg: example; content: something; sid: 18975; rev:1)
编辑：提供的解决方案适用于我在上面提供的初始示例数据，但是它不适用于实际签名。所以我在下面提供了实际的签名。此外，规则在每行之间可能有也可能没有白色-space。
示例文件 1 文本：
#alert tcp $EXTERNAL_NET any -> $HOME_NET 2200 (msg:"ET EXPLOIT CA BrightStor ARCserve Mobile Backup LGSERVER.EXE Heap Corruption"; flow:established,to_server; content:"|4e 3d 2c 1b|"; depth:4; isdataat:2891,relative; reference:cve,2007-0449; reference:url,doc.emergingthreats.net/bin/view/Main/2003369; classtype:attempted-admin; sid:2003369; rev:3; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
alert udp $EXTERNAL_NET any -> $HOME_NET 111 (msg:"ET EXPLOIT Computer Associates Brightstor ARCServer Backup RPC Server (Catirpc.dll) DoS"; content:"|00 00 00 00|"; offset:4; depth:4; content:"|00 00 00 03|"; distance:8; within:4; content:"|00 00 00 08|"; distance:0; within:4; content:"|00 00 00 00|"; distance:0; within:4; content:"|00 00 00 00|"; distance:4; within:4; content:"|00 00 00 00 00 00 00 00|"; distance:8; within:32; reference:url,www.milw0rm.com/exploits/3248; reference:url,doc.emergingthreats.net/bin/view/Main/2003370; classtype:attempted-dos; sid:2003370; rev:3; metadata:created_at 2010_07_30, updated_at 2020_08_20;)
#alert tcp $EXTERNAL_NET any -> $HOME_NET 1900 (msg:"ET EXPLOIT Computer Associates Mobile Backup Service LGSERVER.EXE Stack Overflow"; flow:established,to_server; content:"0000033000"; depth:10; isdataat:1000,relative; reference:url,www.milw0rm.com/exploits/3244; reference:url,doc.emergingthreats.net/bin/view/Main/2003378; classtype:attempted-admin; sid:2003378; rev:3; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
示例文件 2 文本：
#alert tcp $EXTERNAL_NET any -> $HOME_NET 2200 (msg:"ET EXPLOIT CA BrightStor ARCserve Mobile Backup LGSERVER.EXE Heap Corruption"; flow:established,to_server; content:"|4e 3d 2c 1b|"; depth:4; isdataat:2891,relative; reference:cve,2007-0449; reference:url,doc.emergingthreats.net/bin/view/Main/2003369; classtype:attempted-admin; sid:2003369; rev:3; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
alert udp $EXTERNAL_NET any -> $HOME_NET 111 (msg:"ET EXPLOIT Computer Associates Brightstor ARCServer Backup RPC Server (Catirpc.dll) DoS"; content:"|00 00 00 00|"; offset:4; depth:4; content:"|00 00 00 03|"; distance:8; within:4; content:"|00 00 00 08|"; distance:0; within:4; content:"|00 00 00 00|"; distance:0; within:4; content:"|00 00 00 00|"; distance:4; within:4; content:"|00 00 00 00 00 00 00 00|"; distance:8; within:32; reference:url,www.milw0rm.com/exploits/3248; reference:url,doc.emergingthreats.net/bin/view/Main/2003370; classtype:attempted-dos; sid:2003370; rev:3; metadata:created_at 2010_07_30, updated_at 2020_08_20;)
< insert #>alert tcp $EXTERNAL_NET any -> $HOME_NET 1900 (msg:"ET EXPLOIT Computer Associates Mobile Backup Service LGSERVER.EXE Stack Overflow"; flow:established,to_server; content:"0000033000"; depth:10; isdataat:1000,relative; reference:url,www.milw0rm.com/exploits/3244; reference:url,doc.emergingthreats.net/bin/view/Main/2003378; classtype:attempted-admin; sid:2003378; rev:3; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
首先，检查第一个文件，找出哪些 SIDS 被注释掉了：
sed -En '/^#/ s/.*sid:([0-9]+).*//p' file1
以上命令打印出以 # 开头的行的 sid，每行一个 sid。现在让我们汇总这些行并构建一个用 |:
分隔的 sids 列表
sed -En '/^#/ s/.*sid:([0-9]+).*//p' file1 | paste -sd '|'
很好，现在我们有了 sid1|sid2|...|sidN。正如所写，这可以用作正则表达式来识别 file2 中需要注释掉的行。让我们把这个正则表达式放在一个变量中：
sid_regex=$(sed -En '/^#/ s/.*sid:([0-9]+).*//p' file1 | paste -sd '|')
现在，我们可以修改 file2，以便注释掉 1) 具有与正则表达式匹配的 sid 和 2) 尚未以 # 开头的每一行：
sed -E "/sid:($sid_regex);/ s/^[^#]/#&/" file2 > file2.new
瞧！总结一下：
$ sid_regex=$(sed -En '/^#/ s/.*sid:([0-9]+).*//p' file1 | paste -sd '|')
$ sed -E "/sid:($sid_regex);/ s/^[^#]/#&/" file2 > file2.new
[更新] 您有太多注释行，以至于生成的巨大正则表达式使命令太大（“参数列表太长”）。让我们尝试另一种方法：我们将构建多行 sed 程序，而不是使用巨大的正则表达式构建单行 sed 程序，每个 sid 一行。
第一个 sed 命令生成第二个 sed 程序：
sed -En '/^#/ s|.*(sid:[0-9]+;).*|// s/^[^#]/#\&/|p' file1
结果应该是这样的：
/sid:111;/ s/^[^#]/#&/
/sid:222;/ s/^[^#]/#&/
...
/sid:123456;/ s/^[^#]/#&/
现在我们用该程序提供第二个 sed 以处理文件 2:
sed -En '/^#/ s|.*(sid:[0-9]+;).*|// s/^[^#]/#\&/|p' file1 | sed -f - file2 > file2.new
目标：我想比较两个 Suricata 规则文件并注释掉 file1 和 file2 中的相同行（警告“SID”），除非它已经被注释掉。我知道有更好的方法可以使用 Suricata 阈值文件来执行此操作，但不幸的是，我没有超出我可以在这里解释的那种奢侈。这是为了便于更新规则，其中规则可能会更新，但“SID”在两个文件中的共性将相同。
我不知道从哪里开始。
示例文件 1 文本：
alert $home_net any > $External_net any (msg: example; content: something; sid: 12345; rev:1)
#alert $home_net any > $External_net any (msg: example; content: something; sid: 67895; rev:1)
alert $home_net any > $External_net any (msg: example; content: something; sid: 18975; rev:1)
示例文件 2 文本：
alert $home_net any > $External_net any (msg: example; content: something; sid: 12345; rev:1)
<insert #>alert $home_net any > $External_net any (msg: example; content: something; sid: 67895; rev:1)
alert $home_net any > $External_net any (msg: example; content: something; sid: 18975; rev:1)
编辑：提供的解决方案适用于我在上面提供的初始示例数据，但是它不适用于实际签名。所以我在下面提供了实际的签名。此外，规则在每行之间可能有也可能没有白色-space。
示例文件 1 文本：
#alert tcp $EXTERNAL_NET any -> $HOME_NET 2200 (msg:"ET EXPLOIT CA BrightStor ARCserve Mobile Backup LGSERVER.EXE Heap Corruption"; flow:established,to_server; content:"|4e 3d 2c 1b|"; depth:4; isdataat:2891,relative; reference:cve,2007-0449; reference:url,doc.emergingthreats.net/bin/view/Main/2003369; classtype:attempted-admin; sid:2003369; rev:3; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
alert udp $EXTERNAL_NET any -> $HOME_NET 111 (msg:"ET EXPLOIT Computer Associates Brightstor ARCServer Backup RPC Server (Catirpc.dll) DoS"; content:"|00 00 00 00|"; offset:4; depth:4; content:"|00 00 00 03|"; distance:8; within:4; content:"|00 00 00 08|"; distance:0; within:4; content:"|00 00 00 00|"; distance:0; within:4; content:"|00 00 00 00|"; distance:4; within:4; content:"|00 00 00 00 00 00 00 00|"; distance:8; within:32; reference:url,www.milw0rm.com/exploits/3248; reference:url,doc.emergingthreats.net/bin/view/Main/2003370; classtype:attempted-dos; sid:2003370; rev:3; metadata:created_at 2010_07_30, updated_at 2020_08_20;)
#alert tcp $EXTERNAL_NET any -> $HOME_NET 1900 (msg:"ET EXPLOIT Computer Associates Mobile Backup Service LGSERVER.EXE Stack Overflow"; flow:established,to_server; content:"0000033000"; depth:10; isdataat:1000,relative; reference:url,www.milw0rm.com/exploits/3244; reference:url,doc.emergingthreats.net/bin/view/Main/2003378; classtype:attempted-admin; sid:2003378; rev:3; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
示例文件 2 文本：
#alert tcp $EXTERNAL_NET any -> $HOME_NET 2200 (msg:"ET EXPLOIT CA BrightStor ARCserve Mobile Backup LGSERVER.EXE Heap Corruption"; flow:established,to_server; content:"|4e 3d 2c 1b|"; depth:4; isdataat:2891,relative; reference:cve,2007-0449; reference:url,doc.emergingthreats.net/bin/view/Main/2003369; classtype:attempted-admin; sid:2003369; rev:3; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
alert udp $EXTERNAL_NET any -> $HOME_NET 111 (msg:"ET EXPLOIT Computer Associates Brightstor ARCServer Backup RPC Server (Catirpc.dll) DoS"; content:"|00 00 00 00|"; offset:4; depth:4; content:"|00 00 00 03|"; distance:8; within:4; content:"|00 00 00 08|"; distance:0; within:4; content:"|00 00 00 00|"; distance:0; within:4; content:"|00 00 00 00|"; distance:4; within:4; content:"|00 00 00 00 00 00 00 00|"; distance:8; within:32; reference:url,www.milw0rm.com/exploits/3248; reference:url,doc.emergingthreats.net/bin/view/Main/2003370; classtype:attempted-dos; sid:2003370; rev:3; metadata:created_at 2010_07_30, updated_at 2020_08_20;)
< insert #>alert tcp $EXTERNAL_NET any -> $HOME_NET 1900 (msg:"ET EXPLOIT Computer Associates Mobile Backup Service LGSERVER.EXE Stack Overflow"; flow:established,to_server; content:"0000033000"; depth:10; isdataat:1000,relative; reference:url,www.milw0rm.com/exploits/3244; reference:url,doc.emergingthreats.net/bin/view/Main/2003378; classtype:attempted-admin; sid:2003378; rev:3; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
首先，检查第一个文件，找出哪些 SIDS 被注释掉了：
sed -En '/^#/ s/.*sid:([0-9]+).*//p' file1
以上命令打印出以 # 开头的行的 sid，每行一个 sid。现在让我们汇总这些行并构建一个用 |:
sed -En '/^#/ s/.*sid:([0-9]+).*//p' file1 | paste -sd '|'
很好，现在我们有了 sid1|sid2|...|sidN。正如所写，这可以用作正则表达式来识别 file2 中需要注释掉的行。让我们把这个正则表达式放在一个变量中：
sid_regex=$(sed -En '/^#/ s/.*sid:([0-9]+).*//p' file1 | paste -sd '|')
现在，我们可以修改 file2，以便注释掉 1) 具有与正则表达式匹配的 sid 和 2) 尚未以 # 开头的每一行：
sed -E "/sid:($sid_regex);/ s/^[^#]/#&/" file2 > file2.new
瞧！总结一下：
$ sid_regex=$(sed -En '/^#/ s/.*sid:([0-9]+).*//p' file1 | paste -sd '|')
$ sed -E "/sid:($sid_regex);/ s/^[^#]/#&/" file2 > file2.new
[更新] 您有太多注释行，以至于生成的巨大正则表达式使命令太大（“参数列表太长”）。让我们尝试另一种方法：我们将构建多行 sed 程序，而不是使用巨大的正则表达式构建单行 sed 程序，每个 sid 一行。
第一个 sed 命令生成第二个 sed 程序：
sed -En '/^#/ s|.*(sid:[0-9]+;).*|// s/^[^#]/#\&/|p' file1
结果应该是这样的：
/sid:111;/ s/^[^#]/#&/
/sid:222;/ s/^[^#]/#&/
...
/sid:123456;/ s/^[^#]/#&/
现在我们用该程序提供第二个 sed 以处理文件 2:
sed -En '/^#/ s|.*(sid:[0-9]+;).*|// s/^[^#]/#\&/|p' file1 | sed -f - file2 > file2.new