将全角字符转换为半角字符

Question

我有一个程序可以将全角字符转换为半角字符。它工作正常，除了数字零。全角零不转换为半角零。

Perl

use strict;
use warnings;
use warnings qw(FATAL utf8);
use utf8;
use feature qw(unicode_strings);
use open qw(:std :utf8);

unless ( @ARGV == 2 ) { 
    print "Usage: script.pl input_file output_file\n";
    exit;
}

my %fwhw = (
        '０' => '0', '１' => '1', '２' => '2', '３' => '3', '４' => '4',
        '５' => '5', '６' => '6', '７' => '7', '８' => '8', '９' => '9',
        'Ａ' => 'A', 'Ｂ' => 'B', 'Ｃ' => 'C', 'Ｄ' => 'D', 'Ｅ' => 'E',
        'Ｆ' => 'F', 'Ｇ' => 'G', 'Ｈ' => 'H', 'Ｉ' => 'I', 'Ｊ' => 'J',
        'Ｋ' => 'K', 'Ｌ' => 'L', 'Ｍ' => 'M', 'Ｎ' => 'N', 'Ｏ' => 'O',
        'Ｐ' => 'P', 'Ｑ' => 'Q', 'Ｒ' => 'R', 'Ｓ' => 'S', 'Ｔ' => 'T',
        'Ｕ' => 'U', 'Ｖ' => 'V', 'Ｗ' => 'W', 'Ｘ' => 'X', 'Ｙ' => 'Y',
        'Ｚ' => 'Z', 'ａ' => 'a', 'ｂ' => 'b', 'ｃ' => 'c', 'ｄ' => 'd',
        'ｅ' => 'e', 'ｆ' => 'f', 'ｇ' => 'g', 'ｈ' => 'h', 'ｉ' => 'i',
        'ｊ' => 'j', 'ｋ' => 'k', 'ｌ' => 'l', 'ｍ' => 'm', 'ｎ' => 'n',
        'ｏ' => 'o', 'ｐ' => 'p', 'ｑ' => 'q', 'ｒ' => 'r', 'ｓ' => 's',
        'ｔ' => 't', 'ｕ' => 'u', 'ｖ' => 'v', 'ｗ' => 'w', 'ｘ' => 'x',
        'ｙ' => 'y', 'ｚ' => 'z', '－' => '-', '、' => ', ', '　' => ' ',
         '／' => '/',);

sub slurp {
    my $file = shift;
    open my $fh_read, '<', $file or die "Could not open file: $!";
    return do {local $/; <$fh_read>};
}

sub convert {
    my $sub_string = shift;
    $sub_string =~ s/(.)/$fwhw{}?$fwhw{}:/seg;
    return $sub_string;
}

my $string = slurp($ARGV[0]);

$string =~ s/<target>\s*<g id="\d+">\K(.*?)(?=<\/g>\s*<\/target>)/convert()/seg;

open my $fh_write, ">", $ARGV[1] or die "Could not open file: $!";

print $fh_write $string;

close $fh_write;

这是我试过的

我通过检查它们的代码点确定数字 0（零）和字母 O（哦）确实不同。全角 0 为 \x{ff10}。全角字母 O 为 \x{ff2f}。我用这段代码检查了这个：
```
  use Encode;

  sub codepoint_hex {
      sprintf "%04x", ord Encode::decode("UTF-8", shift);
  }

  my $codepoint = codepoint_hex('０');
  print $codepoint, "\n";
```
我检查过哈希确实正确加载了所有键和值。

我还没有尝试过的：

我还没有尝试在 Linux 上复制这个奇怪的东西。我在 Windows 10.

如果有人有任何建议或看到我的错误，我将非常感谢指导。

Answer 1

由于 $fwhw{'０'} returns 0，并且 0 为假，因此不会发生替换。替换

$sub_string =~ s/(.)/$fwhw{}?$fwhw{}:/seg;

和

$sub_string =~ s/(.)/exists($fwhw{})?$fwhw{}:/seg;

如果还是不行，用sprintf "%vX", $str看看你到底有什么。

顺便说一句，

sub convert {
    my $sub_string = shift;
    $sub_string =~ s/(.)/exists($fwhw{})?$fwhw{}:/seg;
    return $sub_string;
}

如果换成

会快很多

sub convert {
    state $chars = join '', keys(%fwhw);
    state $re = qr/([\Q$chars\E])/;
    return $_[0] =~ s/$re/$fwhw{}/gr;
}

更快，

sub convert {
    state $s = join '', keys(%fwhw);
    state $r = join '', values(%fwhw);
    state $tr = eval("sub { $_[0] =~ tr/\Q$s\E/\Q$r\E/r }");
    return $tr->($_[0]);
}

Answer 2

你不需要这么大的词典和这么多的支持功能。一个简单的sed就够了

halfwidth='!"#$%&'\''()*+,-.\/0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~｟｠¢£¬¯¦¥₩ '
fullwidth='！＂＃＄％＆＇（）＊＋，－．／０１２３４５６７８９：；＜＝＞？＠ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ［＼］＾＿｀ａｂｃｄｅｆｇｈｉｊｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ｛｜｝～｟｠￠￡￢￣￤￥￦　'
sed -ie "y/$fullwidth/$halfwidth/" your_file

如果你想在 perl 中这样做也很简单

perl -Mutf8 -i -C -pe 'BEGIN{ use open qw/:std :utf8/; } tr#！＂＃＄％＆＇（）＊＋，－．／０１２３４５６７８９：；＜＝＞？＠ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ［＼］＾＿｀ａｂｃｄｅｆｇｈｉｊｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ｛｜｝～｟｠￠￡￢￣￤￥￦　#!"\#$%&'\''()*+,-.\/0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~｟｠¢£¬¯¦¥₩ # your_file'

将全角字符转换为半角字符

Converting full-width characters to half-width characters

regex

perl

character

data-conversion

cjk

Perl

这是我试过的

我还没有尝试过的：