Ruby 编码 ASCII_8BIT 和扩展 ASCII
Ruby encoding ASCII_8BIT and extended ASCII
关于ASCII_8BIT
Encoding::ASCII_8BIT is a special encoding that is usually used for a byte string, not a character string. But as the name insists, its characters in the range of ASCII are considered as ASCII characters. This is useful when you use ASCII-8BIT characters with other ASCII compatible characters.
上下文
我想使用ASCII_8BIT因为我需要对0x00（0d00）和0xff（0d255）之间的所有字符进行编码，所以ASCII（0-127）加扩展ASCII（128-255）。
ASCII（编码，US-ASCII）是一种 7 位编码，只能识别 ASCII（字符集）字符 (0-127)。顾名思义，我期待 ASCII_8BIT 将其扩展到 8 位以添加对 128-255 的支持。
问题
当我使用 chr 时，编码自动设置为 ASCII_8BIT 但是当我把一个 char 直接放在 128-255 (0x80-0xff) 之间的字符串中，然后询问我得到的编码是什么UTF-8 而不是，如果我尝试将其转换为 ASCII_8BIT 则会出错。
irb(main):014:0> 0x8f.chr
=> "\x8F"
irb(main):015:0> 0x8f.chr.encoding
=> #<Encoding:ASCII-8BIT>
irb(main):016:0> "\x8f".encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
2: from (irb):16
1: from (irb):16:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main):021:0> "\x8F".encoding
=> #<Encoding:UTF-8>
ruby 内核有错误吗？我需要能够对 8
之间的所有内容进行编码
ASCII 8BIT 的另一个名称是 BINARY，因为如前引述所述，它应该能够编码任何字节。
irb(main):035:0> Encoding::ASCII_8BIT.names
=> ["ASCII-8BIT", "BINARY"]
其他编码
请告诉我使用另一种编码不是问题的答案，除非它是一种真正映射所有 255 个扩展 ASCII 字符的编码。
- 我不想使用 UTF-8，因为编码是多字节而不是单字节。
- ISO/IEC 8859-1（Latin1，8 位）仅包含 191 个字符（ASCII + 63 个字符）
One notable way in which ISO character sets differ from code pages is that the character positions 128 to 159, corresponding to ASCII control characters with the high-order bit set, are specifically unused and undefined in the ISO standards, though they had often been used for printable characters in proprietary code pages, a breaking of ISO standards that was almost universal.
Ref. Extended ASCII- ISO 8859 and proprietary adaptations
- Windows-1252（CP-1252，8 位）不包含所有 255 个字符，并且作为扩展 ASCII
的不同映射
ruby 中的可用编码：
irb(main):036:0> Encoding.name_list
=> ["ASCII-8BIT", "UTF-8", "US-ASCII", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "UTF-16", "UTF-32", "UTF8-MAC", "EUC-JP", "Windows-31J", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-KR", "EUC-TW", "GB2312", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "Windows-1250", "Windows-1251", "Windows-1252", "Windows-1253", "Windows-1254", "Windows-1257", "BINARY", "IBM437", "CP437", "IBM737", "CP737", "IBM775", "CP775", "CP850", "IBM850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "CP857", "IBM860", "CP860", "IBM861", "CP861", "IBM862", "CP862", "IBM863", "CP863", "IBM864", "CP864", "IBM865", "CP865", "IBM866", "CP866", "IBM869", "CP869", "Windows-1258", "CP1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "Big5-HKSCS:2008", "CP951", "IBM037", "ebcdic-cp-us", "stateless-ISO-2022-JP", "eucJP", "eucJP-ms", "euc-jp-ms", "CP51932", "EUC-JIS-2004", "EUC-JISX0213", "eucKR", "eucTW", "EUC-CN", "eucCN", "GB12345", "CP936", "ISO-2022-JP", "ISO2022-JP", "ISO-2022-JP-2", "ISO2022-JP2", "CP50220", "CP50221", "ISO8859-1", "ISO8859-2", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "Windows-1256", "CP1256", "ISO8859-7", "ISO8859-8", "Windows-1255", "CP1255", "ISO8859-9", "ISO8859-10", "ISO8859-11", "TIS-620", "Windows-874", "CP874", "ISO8859-13", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "MacJapanese", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "UTF-7", "CP65000", "CP65001", "UTF-8-MAC", "UTF-8-HFS", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP932", "csWindows31J", "SJIS", "PCK", "CP1250", "CP1251", "CP1252", "CP1253", "CP1254", "CP1257", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "locale", "external", "filesystem", "internal"]
用于比较 python 编码 https://docs.python.org/3/library/codecs.html#standard-encodings
注意事项
通过阅读 Extended ASCII - Multi-byte character encodings，似乎唯一真正的扩展 ASCII 编码是 UTF-8，但它是 Multi-byte 。似乎也不存在真正的扩展 ASCII 单字节编码。
从字节的角度来看，我可以使用此处所述的任何 8 位（单字节）编码 Extended ASCII - Usage in computer-readable languages
all ASCII bytes (0x00 to 0x7F) have the same meaning in all variants of extended ASCII,
但问题是像 ISO-8859-1 这样的实现特别未定义一些字符范围，因此会以错误结束。
irb(main):009:0> (0..255).map { |c| c.chr}.join.encode(Encoding::ISO_8859_1)
Traceback (most recent call last):
6: from /usr/bin/irb:23:in `<main>'
5: from /usr/bin/irb:23:in `load'
4: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
3: from (irb):9
2: from (irb):9:in `rescue in irb_binding'
1: from (irb):9:in `encode'
Encoding::UndefinedConversionError ("\x80" to UTF-8 in conversion from ASCII-8BIT to UTF-8 to ISO-8859-1)
更新 - force_encoding
我找到了字符串方法 force_encoding。
irb(main)> a = "\x8f"
=> "\x8F"
irb(main)> a.encoding
=> #<Encoding:UTF-8>
irb(main)> a.encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
2: from (irb):42
1: from (irb):42:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main)> a.force_encoding(Encoding::ASCII_8BIT)
=> "\x8F"
irb(main):040:0> a.encoding
=> #<Encoding:ASCII-8BIT>
使用 force_encoding 而不是 encode 有什么危险？只是如果我不小心传递了一个多字节字符，它会被转换为多个单字节字符吗？因此，如果确保传递给应用程序的所有字符都在扩展的 ASCII 范围（单字节）内但不安全并且如果将某些 UTF-8 字符传递给应用程序将导致静默转换问题，那么这并不危险。
irb(main):044:0> "\ud087".force_encoding(Encoding::ASCII_8BIT)
=> "\xED\x82\x87"
irb(main):045:0> "\ud087".bytes
=> [237, 130, 135]
更新 - 答案
@mu-is-too-short 的回答和@ForeverZer0 的评论暗示我应该使用 pack 和 unpack 来处理原始字节。
所以与其使用编码和解决方法不如使用它
pattern = 'A' * 2606 + "\x8F\x35\x4A\x5F" + 'C' * 390
pattern.force_encoding(Encoding::ASCII_8BIT)
我应该直接使用bytes
pattern = ['A'.ord] * 2606 + [0x8F, 0x35, 0x4A, 0x5F] + ['C'.ord] * 390
pattern = pattern.pack('C*')
或者这个更容易阅读的语法
pattern = 'A'.bytes * 2606 + "\x8F\x35\x4A\x5F".bytes + 'C'.bytes * 390
pattern = pattern.pack('C*')
无论字节是否为有效的 UTF-8，字符串文字（通常）都是 UTF-8 编码的。因此：
"\x8f".encoding
即使字符串不是有效的 UTF-8，也说 UTF-8。您应该安全地使用 String#force_encoding but if you really want to work with raw bytes, you might be better of working with arrays of integers and using Array#pack 将它们混合成字符串：
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*')
# "\x8F\x11\x06#\xFF\x00"
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*').encoding
# #<Encoding:ASCII-8BIT>
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*').bytes
# [143, 17, 6, 35, 255, 0]
结果应该是相同的，但是，IMO，这是明确使用二进制数据（即原始字节），使您的意图明确，并且应该避免任何编码问题。
还有 String#unpack 如果您正在读取的字节存在已知结构并且您想破解它。
如果您正在进行原始数据包操作，那么所有内容都必须处于 BINARY / ASCII_8BIT 模式，因为 它不是文本 并且不应被处理因此。如果你有除此之外的任何编码 Ruby 将尝试转换它，在最好的情况下会严重破坏二进制数据，在最坏的情况下会因为转换错误而崩溃。
在Ruby术语中，ASCII_8BIT实际上是一个原始数据缓冲区。
代码中字符串的默认编码是 UTF-8：
p "example".encoding
# => #<Encoding:UTF-8>
您可以使用 # encoding: BINARY:
为内联字符串 per-file 设置 Ruby 编码
# encoding: BINARY
p "example".encoding
# => #<Encoding:ASCII-8BIT>
正如 mu 指出的那样，使用 pack 之类的工具来表达二进制数据通常会更好，因为在这种情况下您不会弄错，而且您一开始就没有真正使用字符串。这是非常重要的，因为 8 位值很容易处理，但是 16 位和 32 位值 必须 正确编码，所以你会经常看到很多这样的：
header = pack('nn', qtype, qclass)
其中包含一个涉及两个 16 位值的 DNS header。
关于ASCII_8BIT
Encoding::ASCII_8BIT is a special encoding that is usually used for a byte string, not a character string. But as the name insists, its characters in the range of ASCII are considered as ASCII characters. This is useful when you use ASCII-8BIT characters with other ASCII compatible characters.
上下文
我想使用ASCII_8BIT因为我需要对0x00（0d00）和0xff（0d255）之间的所有字符进行编码，所以ASCII（0-127）加扩展ASCII（128-255）。 ASCII（编码，US-ASCII）是一种 7 位编码，只能识别 ASCII（字符集）字符 (0-127)。顾名思义，我期待 ASCII_8BIT 将其扩展到 8 位以添加对 128-255 的支持。
问题
当我使用 chr 时，编码自动设置为 ASCII_8BIT 但是当我把一个 char 直接放在 128-255 (0x80-0xff) 之间的字符串中，然后询问我得到的编码是什么UTF-8 而不是，如果我尝试将其转换为 ASCII_8BIT 则会出错。
irb(main):014:0> 0x8f.chr
=> "\x8F"
irb(main):015:0> 0x8f.chr.encoding
=> #<Encoding:ASCII-8BIT>
irb(main):016:0> "\x8f".encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
2: from (irb):16
1: from (irb):16:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main):021:0> "\x8F".encoding
=> #<Encoding:UTF-8>
ruby 内核有错误吗？我需要能够对 8
之间的所有内容进行编码ASCII 8BIT 的另一个名称是 BINARY，因为如前引述所述，它应该能够编码任何字节。
irb(main):035:0> Encoding::ASCII_8BIT.names
=> ["ASCII-8BIT", "BINARY"]
其他编码
请告诉我使用另一种编码不是问题的答案，除非它是一种真正映射所有 255 个扩展 ASCII 字符的编码。
- 我不想使用 UTF-8，因为编码是多字节而不是单字节。
- ISO/IEC 8859-1（Latin1，8 位）仅包含 191 个字符（ASCII + 63 个字符）
One notable way in which ISO character sets differ from code pages is that the character positions 128 to 159, corresponding to ASCII control characters with the high-order bit set, are specifically unused and undefined in the ISO standards, though they had often been used for printable characters in proprietary code pages, a breaking of ISO standards that was almost universal. Ref. Extended ASCII- ISO 8859 and proprietary adaptations
- Windows-1252（CP-1252，8 位）不包含所有 255 个字符，并且作为扩展 ASCII 的不同映射
ruby 中的可用编码：
irb(main):036:0> Encoding.name_list
=> ["ASCII-8BIT", "UTF-8", "US-ASCII", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "UTF-16", "UTF-32", "UTF8-MAC", "EUC-JP", "Windows-31J", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-KR", "EUC-TW", "GB2312", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "Windows-1250", "Windows-1251", "Windows-1252", "Windows-1253", "Windows-1254", "Windows-1257", "BINARY", "IBM437", "CP437", "IBM737", "CP737", "IBM775", "CP775", "CP850", "IBM850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "CP857", "IBM860", "CP860", "IBM861", "CP861", "IBM862", "CP862", "IBM863", "CP863", "IBM864", "CP864", "IBM865", "CP865", "IBM866", "CP866", "IBM869", "CP869", "Windows-1258", "CP1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "Big5-HKSCS:2008", "CP951", "IBM037", "ebcdic-cp-us", "stateless-ISO-2022-JP", "eucJP", "eucJP-ms", "euc-jp-ms", "CP51932", "EUC-JIS-2004", "EUC-JISX0213", "eucKR", "eucTW", "EUC-CN", "eucCN", "GB12345", "CP936", "ISO-2022-JP", "ISO2022-JP", "ISO-2022-JP-2", "ISO2022-JP2", "CP50220", "CP50221", "ISO8859-1", "ISO8859-2", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "Windows-1256", "CP1256", "ISO8859-7", "ISO8859-8", "Windows-1255", "CP1255", "ISO8859-9", "ISO8859-10", "ISO8859-11", "TIS-620", "Windows-874", "CP874", "ISO8859-13", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "MacJapanese", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "UTF-7", "CP65000", "CP65001", "UTF-8-MAC", "UTF-8-HFS", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP932", "csWindows31J", "SJIS", "PCK", "CP1250", "CP1251", "CP1252", "CP1253", "CP1254", "CP1257", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "locale", "external", "filesystem", "internal"]
用于比较 python 编码 https://docs.python.org/3/library/codecs.html#standard-encodings
注意事项
通过阅读 Extended ASCII - Multi-byte character encodings，似乎唯一真正的扩展 ASCII 编码是 UTF-8，但它是 Multi-byte 。似乎也不存在真正的扩展 ASCII 单字节编码。
从字节的角度来看，我可以使用此处所述的任何 8 位（单字节）编码 Extended ASCII - Usage in computer-readable languages
all ASCII bytes (0x00 to 0x7F) have the same meaning in all variants of extended ASCII,
但问题是像 ISO-8859-1 这样的实现特别未定义一些字符范围，因此会以错误结束。
irb(main):009:0> (0..255).map { |c| c.chr}.join.encode(Encoding::ISO_8859_1)
Traceback (most recent call last):
6: from /usr/bin/irb:23:in `<main>'
5: from /usr/bin/irb:23:in `load'
4: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
3: from (irb):9
2: from (irb):9:in `rescue in irb_binding'
1: from (irb):9:in `encode'
Encoding::UndefinedConversionError ("\x80" to UTF-8 in conversion from ASCII-8BIT to UTF-8 to ISO-8859-1)
更新 - force_encoding
我找到了字符串方法 force_encoding。
irb(main)> a = "\x8f"
=> "\x8F"
irb(main)> a.encoding
=> #<Encoding:UTF-8>
irb(main)> a.encode(Encoding::ASCII_8BIT)
Traceback (most recent call last):
5: from /usr/bin/irb:23:in `<main>'
4: from /usr/bin/irb:23:in `load'
3: from /usr/lib/ruby/gems/2.6.0/gems/irb-1.0.0/exe/irb:11:in `<top (required)>'
2: from (irb):42
1: from (irb):42:in `encode'
Encoding::InvalidByteSequenceError ("\x8F" on UTF-8)
irb(main)> a.force_encoding(Encoding::ASCII_8BIT)
=> "\x8F"
irb(main):040:0> a.encoding
=> #<Encoding:ASCII-8BIT>
使用 force_encoding 而不是 encode 有什么危险？只是如果我不小心传递了一个多字节字符，它会被转换为多个单字节字符吗？因此，如果确保传递给应用程序的所有字符都在扩展的 ASCII 范围（单字节）内但不安全并且如果将某些 UTF-8 字符传递给应用程序将导致静默转换问题，那么这并不危险。
irb(main):044:0> "\ud087".force_encoding(Encoding::ASCII_8BIT)
=> "\xED\x82\x87"
irb(main):045:0> "\ud087".bytes
=> [237, 130, 135]
更新 - 答案
@mu-is-too-short 的回答和@ForeverZer0 的评论暗示我应该使用 pack 和 unpack 来处理原始字节。
所以与其使用编码和解决方法不如使用它
pattern = 'A' * 2606 + "\x8F\x35\x4A\x5F" + 'C' * 390
pattern.force_encoding(Encoding::ASCII_8BIT)
我应该直接使用bytes
pattern = ['A'.ord] * 2606 + [0x8F, 0x35, 0x4A, 0x5F] + ['C'.ord] * 390
pattern = pattern.pack('C*')
或者这个更容易阅读的语法
pattern = 'A'.bytes * 2606 + "\x8F\x35\x4A\x5F".bytes + 'C'.bytes * 390
pattern = pattern.pack('C*')
无论字节是否为有效的 UTF-8，字符串文字（通常）都是 UTF-8 编码的。因此：
"\x8f".encoding
即使字符串不是有效的 UTF-8，也说 UTF-8。您应该安全地使用 String#force_encoding but if you really want to work with raw bytes, you might be better of working with arrays of integers and using Array#pack 将它们混合成字符串：
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*')
# "\x8F\x11\x06#\xFF\x00"
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*').encoding
# #<Encoding:ASCII-8BIT>
[ 0x8f, 0x11, 0x06, 0x23, 0xff, 0x00 ].pack('C*').bytes
# [143, 17, 6, 35, 255, 0]
结果应该是相同的，但是，IMO，这是明确使用二进制数据（即原始字节），使您的意图明确，并且应该避免任何编码问题。
还有 String#unpack 如果您正在读取的字节存在已知结构并且您想破解它。
如果您正在进行原始数据包操作，那么所有内容都必须处于 BINARY / ASCII_8BIT 模式，因为 它不是文本 并且不应被处理因此。如果你有除此之外的任何编码 Ruby 将尝试转换它，在最好的情况下会严重破坏二进制数据，在最坏的情况下会因为转换错误而崩溃。
在Ruby术语中，ASCII_8BIT实际上是一个原始数据缓冲区。
代码中字符串的默认编码是 UTF-8：
p "example".encoding
# => #<Encoding:UTF-8>
您可以使用 # encoding: BINARY:
# encoding: BINARY
p "example".encoding
# => #<Encoding:ASCII-8BIT>
正如 mu 指出的那样，使用 pack 之类的工具来表达二进制数据通常会更好，因为在这种情况下您不会弄错，而且您一开始就没有真正使用字符串。这是非常重要的，因为 8 位值很容易处理，但是 16 位和 32 位值 必须 正确编码，所以你会经常看到很多这样的：
header = pack('nn', qtype, qclass)
其中包含一个涉及两个 16 位值的 DNS header。