在 perl 中比较外国字符集的字符串的更快方法

Question

我有以下运行良好的代码：

foreach my $type (qw/Arabic Armenian Bengali Bopomofo Braille Buhid Canadian_Aboriginal Cherokee Cyrillic Devanagari Ethiopic Georgian Greek Gujarati Gurmukhi Han Hangul Hanunoo Hebrew Hiragana Kannada Katakana Khmer Lao Limbu Malayalam Mongolian Myanmar Ogham Oriya Runic Sinhala Syriac Tagalog Tagbanwa TaiLe Tamil Telugu Thaana Thai Tibetan/) {
    if ($page_title =~ /\p{Script_Extensions=$type}/i) {

        print qq|TITLE: $_->{domain} is not english ($type), so lets ignore it...\n| if $DEBUG > 0;

        last;
    }
}

它所做的只是寻找特定的字符集，这样我们就可以摆脱那些我们不想要的。现在虽然它可以工作，但它有点慢（因为它对每个都做了 foreach()）。在单个正则表达式中有没有办法做到这一点？（如果可能，还提取匹配集）

更新： 我现在按照建议尝试：

if ($page_title =~ /\p{Script_Extensions=Han|Arabic|Armenian|Bengali|Bopomofo|Braille|Buhid|Canadian_Aboriginal|Cherokee|Cyrillic|Devanagari|Ethiopic|Georgian|Greek|Gujarati|Gurmukhi|Hangul|Hanunoo|Hebrew|Hiragana|Kannada|Katakana|Khmer|Lao|Limbu|Malayalam|Mongolian|Myanmar|Ogham|Oriya|Runic|Sinhala|Syriac|Tagalog|Tagbanwa|TaiLe|Tamil|Telugu|Thaana|Thai|Tibetan}/i) {
    colored(qq|$page_title matches , so lets ignore... |, 'yellow on_magenta'), "\n";
}

还有：

if ($page_title =~ /\p{Script_Extensions=(Han|Arabic|Armenian|Bengali|Bopomofo|Braille|Buhid|Canadian_Aboriginal|Cherokee|Cyrillic|Devanagari|Ethiopic|Georgian|Greek|Gujarati|Gurmukhi|Hangul|Hanunoo|Hebrew|Hiragana|Kannada|Katakana|Khmer|Lao|Limbu|Malayalam|Mongolian|Myanmar|Ogham|Oriya|Runic|Sinhala|Syriac|Tagalog|Tagbanwa|TaiLe|Tamil|Telugu|Thaana|Thai|Tibetan)}/i) {
    colored(qq|$page_title matches , so lets ignore... |, 'yellow on_magenta'), "\n";
}

但是我得到一个错误：

Can't find Unicode property definition "Script_Extensions=Han|Arabic|Armenian|Bengali|Bopomofo|Braille|Buhid|Canadian_Aboriginal|Cherokee|Cyrillic|Devanagari|Ethiopic|Georgian|Greek|Gujarati|Gurmukhi|Hangul|Hanunoo|Hebrew|Hiragana|Kannada|Katakana|Khmer|Lao|Limbu|Malayalam|Mongolian|Myanmar|Ogham|Oriya|Runic|Sinhala|Syriac|Tagalog|Tagbanwa|TaiLe|Tamil|Telugu|Thaana|Thai|Tibetan" in regex; marked by <-- HERE in m/\p{Script_Extensions=Han|Arabic|Armenian|Bengali|Bopomofo|Braille|Buhid|Canadian_Aboriginal|Che at process-all-domains.cgi line 300.

Answer 1

我似乎可以让它工作的唯一方法是：

if ($page_title =~ /\p{Script_Extensions=Han}|\p{Script_Extensions=Arabic}|\p{Script_Extensions=Armenian}|\p{Script_Extensions=Bengali}|\p{Script_Extensions=Bopomofo}|\p{Script_Extensions=Braille}|\p{Script_Extensions=Buhid}|\p{Script_Extensions=Canadian_Aboriginal}|\p{Script_Extensions=Cherokee}|\p{Script_Extensions=Cyrillic}|\p{Script_Extensions=Devanagari}|\p{Script_Extensions=Ethiopic}|\p{Script_Extensions=Georgian}|\p{Script_Extensions=Greek}|\p{Script_Extensions=Gujarati}|\p{Script_Extensions=Gurmukhi}|\p{Script_Extensions=Hangul}|\p{Script_Extensions=Hanunoo}|\p{Script_Extensions=Hebrew}|\p{Script_Extensions=Hiragana}|\p{Script_Extensions=Kannada}|\p{Script_Extensions=Katakana}|\p{Script_Extensions=Khmer}|\p{Script_Extensions=Lao}|\p{Script_Extensions=Limbu}|\p{Script_Extensions=Malayalam}|\p{Script_Extensions=Mongolian}|\p{Script_Extensions=Myanmar}|\p{Script_Extensions=Ogham}|\p{Script_Extensions=Oriya}|\p{Script_Extensions=Runic}|\p{Script_Extensions=Sinhala}|\p{Script_Extensions=Syriac}|\p{Script_Extensions=Tagalog}|\p{Script_Extensions=Tagbanwa}|\p{Script_Extensions=TaiLe}|\p{Script_Extensions=Tamil}|\p{Script_Extensions=Telugu}|\p{Script_Extensions=Thaana}|\p{Script_Extensions=Thai}|\p{Script_Extensions=Tibetan}/i) {
    # ... match
}

它有点乱，但据我所知它正在运行。

基准测试示例：

  use Benchmark;

  my $page = Common::get_html_file("bsscn2p.com","home");

  my $page_title;
  if ($page =~ /\<title\>(.+?)\<\/title\>/i) {
    $page_title = ;
  }

  timethese(100000, {
    test1 => sub {
      if ($page_title =~ /\p{Script_Extensions=Han}|\p{Script_Extensions=Arabic}|\p{Script_Extensions=Armenian}|\p{Script_Extensions=Bengali}|\p{Script_Extensions=Bopomofo}|\p{Script_Extensions=Braille}|\p{Script_Extensions=Buhid}|\p{Script_Extensions=Canadian_Aboriginal}|\p{Script_Extensions=Cherokee}|\p{Script_Extensions=Cyrillic}|\p{Script_Extensions=Devanagari}|\p{Script_Extensions=Ethiopic}|\p{Script_Extensions=Georgian}|\p{Script_Extensions=Greek}|\p{Script_Extensions=Gujarati}|\p{Script_Extensions=Gurmukhi}|\p{Script_Extensions=Hangul}|\p{Script_Extensions=Hanunoo}|\p{Script_Extensions=Hebrew}|\p{Script_Extensions=Hiragana}|\p{Script_Extensions=Kannada}|\p{Script_Extensions=Katakana}|\p{Script_Extensions=Khmer}|\p{Script_Extensions=Lao}|\p{Script_Extensions=Limbu}|\p{Script_Extensions=Malayalam}|\p{Script_Extensions=Mongolian}|\p{Script_Extensions=Myanmar}|\p{Script_Extensions=Ogham}|\p{Script_Extensions=Oriya}|\p{Script_Extensions=Runic}|\p{Script_Extensions=Sinhala}|\p{Script_Extensions=Syriac}|\p{Script_Extensions=Tagalog}|\p{Script_Extensions=Tagbanwa}|\p{Script_Extensions=TaiLe}|\p{Script_Extensions=Tamil}|\p{Script_Extensions=Telugu}|\p{Script_Extensions=Thaana}|\p{Script_Extensions=Thai}|\p{Script_Extensions=Tibetan}/i) {
        #print qq|TITLE: $page_title matches, so lets ignore... $_->{domain} \n|;
      }
    },
    test2 => sub {

      foreach my $type (qw/Han Arabic Armenian Bengali Bopomofo Braille Buhid Canadian_Aboriginal Cherokee Cyrillic Devanagari Ethiopic Georgian Greek Gujarati Gurmukhi Hangul Hanunoo Hebrew Hiragana Kannada Katakana Khmer Lao Limbu Malayalam Mongolian Myanmar Ogham Oriya Runic Sinhala Syriac Tagalog Tagbanwa TaiLe Tamil Telugu Thaana Thai Tibetan/) {
          if ($page_title =~ /\p{Script_Extensions=$type}/i) {
            #print "MATCH for $type! \n";
          }
      }

    }
  });


Benchmark: timing 100000 iterations of test1, test2...
     test1:  0 wallclock secs ( 0.03 usr +  0.00 sys =  0.03 CPU) @ 3333333.33/s (n=100000)
            (warning: too few iterations for a reliable count)
     test2: 117 wallclock secs (115.36 usr +  0.13 sys = 115.49 CPU) @ 865.88/s (n=100000)

使用新代码会更好更高效 :)

Answer 2

首先，我猜想使用 /i 没有意义。

至于解决方案，交替是一种选择。

/
     \p{Script_Extensions=Arabic}
   | \p{Script_Extensions=Armenian}
   | \p{Script_Extensions=Bengali}
   | ...
   | \p{Script_Extensions=Thaana}
   | \p{Script_Extensions=Thai}
   | \p{Script_Extensions=Tibetan}
/x

交替允许我们使用空格，但更快的解决方案是字符 class.

/
   [\p{Script_Extensions=Arabic}\p{Script_Extensions=Armenian}\p{Script_Extensions=Bengali}...\p{Script_Extensions=Thaana}\p{Script_Extensions=Thai}\p{Script_Extensions=Tibetan}]
/x

但是，41 个属性中只有 3 个占据了整个屏幕宽度。与此处介绍的其他解决方案一样，没有什么能阻止您动态构建模式。

my $class_body =
   join '',
      map "\p{Script_Extensions=$_}",
         qw(
            Arabic Armenian Bengali
            ...
            Thaana Thai Tibetan
         );

/[$class_body]/

但还有一个选择：(?[...])

use experimental qw( regex_sets );

/
   (?[ \p{Script_Extensions=Arabic}
   +   \p{Script_Extensions=Armenian}
   +   \p{Script_Extensions=Bengali}
   +   ...
   +   \p{Script_Extensions=Thaana}
   +   \p{Script_Extensions=Thai}
   +   \p{Script_Extensions=Tibetan}
   ])
/x

它仍处于实验阶段，但我认为它很稳定并且很可能将其变成语言。

在 perl 中比较外国字符集的字符串的更快方法

Faster way to do this comparison of string in perl for foreign charsets

perl