从我的 'cat' 中排除单行，保留多行相似的行

Question

这是我的问题，我有一个这样的脚本命令：

cat /home/tmp/stats* | grep "test" | grep 'Default;;' | sort

2020-01-26 11:01:31;10433;SERVICES;test;_AVaHYDcvqbshjdkzah3w;0.1;Default;;end;success;90231
2020-01-26 11:15:01;19260;SERVICES;test;_AVaHYDcvqbshjdkzah3w;0.1;Default;;begin;;
2020-01-26 11:15:54;19260;SERVICES;test;_AVaHYDcvqbshjdkzah3w;0.1;Default;;end;success;53294
2020-01-26 11:30:02;25161;SERVICES;test;_AVaHYDcvqbshjdkzah3w;0.1;Default;;begin;;
2020-01-26 11:31:16;25161;SERVICES;test;_AVaHYDcvqbshjdkzah3w;0.1;Default;;end;success;74425

我想添加一个警告，即 "if you encounter the same code (10433, 19260, 25161) less than twice, then you exclude it from the list."

像这样：

cat /home/tmp/stats* | grep "test" | grep 'Default;;' | [if the code appears only once, exclude] | sort

2020-01-26 11:15:01;19260;SERVICES;test;_AVaHYDcvqbshjdkzah3w;0.1;Default;;begin;;
2020-01-26 11:15:54;19260;SERVICES;test;_AVaHYDcvqbshjdkzah3w;0.1;Default;;end;success;53294
2020-01-26 11:30:02;25161;SERVICES;test;_AVaHYDcvqbshjdkzah3w;0.1;Default;;begin;;
2020-01-26 11:31:16;25161;SERVICES;test;_AVaHYDcvqbshjdkzah3w;0.1;Default;;end;success;74425

这可以在一行中实现吗？如果没有，我该怎么做？

提前致谢。

Answer 1

第一个解决方案（通过 OP 的尝试）： 能否请您尝试以下操作。

your_command | awk 'BEGIN{FS=";"} {a[]++;b[]=(b[]?b[] ORS:"")[=10=]} END{for(i in a){if(a[i]>=2){print b[i]}}}'

说明：为以上代码添加详细说明。

your_command |                     ##Sending OP command output to next awk command.
awk '                              ##Starting awk command from here.
BEGIN{                             ##Starting BEGIN section from here.
  FS=";"                           ##Setting FS(field separator) as semi-colon here.
}
{
  a[]++                          ##Creating an array a with index (2nd field of current line) with increasing value by 1 each time it comes here.
  b[]=(b[]?b[] ORS:"")[=11=]     ##Creating an array b with index  and keep concatenating its own value in it.
}
END{                               ##Starting END block for this awk program from here.
  for(i in a){                     ##Traversing thorough array a here.
    if(a[i]>=2){                   ##Checking condition if value of array a with index i is greater than or equal to 2.
      print b[i]                   ##Printing value of array b with index i here.
    }
  }
}
'

第二个解决方案（尝试在单个命令本身中实现）： 或尝试在单个命令中实现看到 OP 的尝试：

awk '
BEGIN{
  FS=";"
}
!/test|default/{
  next
}
{
  a[]++
  b[]=(b[]?b[] ORS:"")[=12=]
}
END{
  for(i in a){
    if(a[i]>=2){
      print b[i]
    }
  }
}
' /home/tmp/stats*

从我的 'cat' 中排除单行，保留多行相似的行

Exclude single line from my 'cat', keep multiple similar lines

awk

cat