如何在 Android 9 上保持 UTF-8 的向后兼容性?
How to keep backward compatibility of UTF-8 on Android 9?
Android 9 中引入的行为更改之一更加严格 UTF-8 decoder。如果我们有一个不正确的 UTF-8 字符串字节数组(例如随机字节或一些二进制数据)并尝试从中创建一个字符串:
return new String(bytes)
Android 将选择 UTF-8 作为首选编码(这很好)但是 return Android 9 上的结果与旧版本 [=34= 上的结果略有不同].
我知道将随机字节转换为 UTF-8 字符串首先听起来不是个好主意,但我现在需要向后兼容性。
是否可以选择在所有 Android 版本上获得完全相同的字符串结果?
编辑:
重现步骤:
byte[] bytes = new byte[]{25, 17, 113, 18, 62, 121, -6, -71, 45, -126, -113, 122, 58, 49, -30, -53, -66, -7, 0, -41};
char[] password = new String(bytes).toCharArray();
byte[] passKey = PBEParametersGenerator.PKCS12PasswordToBytes(password);
Log.d("TEST", "Bytes: ".concat(Arrays.toString(passKey)));
Android <9.0:
的输出
[0, 25, 0, 17, 0, 113, 0, 18, 0, 62, 0, 121, -1, -3, 0, 45, -1, -3, -1, -3, 0, 122, 0, 58, 0, 49, -1, -3, 2, -2, -1, -3, 0, 0, -1, -3, 0, 0]
Android9.0 的输出:
[0, 25, 0, 17, 0, 113, 0, 18, 0, 62, 0, 121, -1, -3, -1, -3, 0, 45, -1, -3, -1, -3, 0, 122, 0, 58, 0, 49, -1, -3, 2, -2, -1, -3, 0, 0, -1, -3, 0, 0]
您可以使用此代码,它是从以前的 Android 版本的 UTF-8 解码器移植而来的。
private static final char REPLACEMENT_CHAR = (char) 0xfffd;
public static char[] byteArrayToCharArray(byte[] data) {
char[] value;
final int offset = 0;
final int byteCount = data.length;
char[] v = new char[byteCount];
int idx = offset;
int last = offset + byteCount;
int s = 0;
outer:
while (idx < last) {
byte b0 = data[idx++];
if ((b0 & 0x80) == 0) {
// 0xxxxxxx
// Range: U-00000000 - U-0000007F
int val = b0 & 0xff;
v[s++] = (char) val;
} else if (((b0 & 0xe0) == 0xc0) || ((b0 & 0xf0) == 0xe0) ||
((b0 & 0xf8) == 0xf0) || ((b0 & 0xfc) == 0xf8) || ((b0 & 0xfe) == 0xfc)) {
int utfCount = 1;
if ((b0 & 0xf0) == 0xe0) utfCount = 2;
else if ((b0 & 0xf8) == 0xf0) utfCount = 3;
else if ((b0 & 0xfc) == 0xf8) utfCount = 4;
else if ((b0 & 0xfe) == 0xfc) utfCount = 5;
// 110xxxxx (10xxxxxx)+
// Range: U-00000080 - U-000007FF (count == 1)
// Range: U-00000800 - U-0000FFFF (count == 2)
// Range: U-00010000 - U-001FFFFF (count == 3)
// Range: U-00200000 - U-03FFFFFF (count == 4)
// Range: U-04000000 - U-7FFFFFFF (count == 5)
if (idx + utfCount > last) {
v[s++] = REPLACEMENT_CHAR;
continue;
}
// Extract usable bits from b0
int val = b0 & (0x1f >> (utfCount - 1));
for (int i = 0; i < utfCount; ++i) {
byte b = data[idx++];
if ((b & 0xc0) != 0x80) {
v[s++] = REPLACEMENT_CHAR;
idx--; // Put the input char back
continue outer;
}
// Push new bits in from the right side
val <<= 6;
val |= b & 0x3f;
}
// Note: Java allows overlong char
// specifications To disallow, check that val
// is greater than or equal to the minimum
// value for each count:
//
// count min value
// ----- ----------
// 1 0x80
// 2 0x800
// 3 0x10000
// 4 0x200000
// 5 0x4000000
// Allow surrogate values (0xD800 - 0xDFFF) to
// be specified using 3-byte UTF values only
if ((utfCount != 2) && (val >= 0xD800) && (val <= 0xDFFF)) {
v[s++] = REPLACEMENT_CHAR;
continue;
}
// Reject chars greater than the Unicode maximum of U+10FFFF.
if (val > 0x10FFFF) {
v[s++] = REPLACEMENT_CHAR;
continue;
}
// Encode chars from U+10000 up as surrogate pairs
if (val < 0x10000) {
v[s++] = (char) val;
} else {
int x = val & 0xffff;
int u = (val >> 16) & 0x1f;
int w = (u - 1) & 0xffff;
int hi = 0xd800 | (w << 6) | (x >> 10);
int lo = 0xdc00 | (x & 0x3ff);
v[s++] = (char) hi;
v[s++] = (char) lo;
}
} else {
// Illegal values 0x8*, 0x9*, 0xa*, 0xb*, 0xfd-0xff
v[s++] = REPLACEMENT_CHAR;
}
}
if (s == byteCount) {
// We guessed right, so we can use our temporary array as-is.
value = v;
} else {
// Our temporary array was too big, so reallocate and copy.
value = new char[s];
System.arraycopy(v, 0, value, 0, s);
}
return value;
}
我通过创建一个 20 字节的随机数组对此进行了单元测试,并与之前 Android 版本的原始 new String(bytes).toCharArray()
实现进行了比较,然后重复了一百万次。我没有看到多个旧 Android 版本有任何差异。
原始源代码来自这里:https://android.googlesource.com/platform/libcore/+/a7752f4d22097346dd7849b92b9f36d0a0a7a8f3/libdvm/src/main/java/java/lang/String.java#245
为了简化它,我删除了处理非UTF8字符集的部分,如果你使用new String()
,那么UTF-8是使用的默认字符集,所以你应该没问题。
此代码将使您获得所需的向后兼容性。但是正如其他人会建议你的那样,我建议尽可能寻找一个更简单的解决方案,它不依赖于 Android 版本(或其他不受你控制的组件)
Android 9 中引入的行为更改之一更加严格 UTF-8 decoder。如果我们有一个不正确的 UTF-8 字符串字节数组(例如随机字节或一些二进制数据)并尝试从中创建一个字符串:
return new String(bytes)
Android 将选择 UTF-8 作为首选编码(这很好)但是 return Android 9 上的结果与旧版本 [=34= 上的结果略有不同].
我知道将随机字节转换为 UTF-8 字符串首先听起来不是个好主意,但我现在需要向后兼容性。
是否可以选择在所有 Android 版本上获得完全相同的字符串结果?
编辑:
重现步骤:
byte[] bytes = new byte[]{25, 17, 113, 18, 62, 121, -6, -71, 45, -126, -113, 122, 58, 49, -30, -53, -66, -7, 0, -41};
char[] password = new String(bytes).toCharArray();
byte[] passKey = PBEParametersGenerator.PKCS12PasswordToBytes(password);
Log.d("TEST", "Bytes: ".concat(Arrays.toString(passKey)));
Android <9.0:
的输出[0, 25, 0, 17, 0, 113, 0, 18, 0, 62, 0, 121, -1, -3, 0, 45, -1, -3, -1, -3, 0, 122, 0, 58, 0, 49, -1, -3, 2, -2, -1, -3, 0, 0, -1, -3, 0, 0]
Android9.0 的输出:
[0, 25, 0, 17, 0, 113, 0, 18, 0, 62, 0, 121, -1, -3, -1, -3, 0, 45, -1, -3, -1, -3, 0, 122, 0, 58, 0, 49, -1, -3, 2, -2, -1, -3, 0, 0, -1, -3, 0, 0]
您可以使用此代码,它是从以前的 Android 版本的 UTF-8 解码器移植而来的。
private static final char REPLACEMENT_CHAR = (char) 0xfffd;
public static char[] byteArrayToCharArray(byte[] data) {
char[] value;
final int offset = 0;
final int byteCount = data.length;
char[] v = new char[byteCount];
int idx = offset;
int last = offset + byteCount;
int s = 0;
outer:
while (idx < last) {
byte b0 = data[idx++];
if ((b0 & 0x80) == 0) {
// 0xxxxxxx
// Range: U-00000000 - U-0000007F
int val = b0 & 0xff;
v[s++] = (char) val;
} else if (((b0 & 0xe0) == 0xc0) || ((b0 & 0xf0) == 0xe0) ||
((b0 & 0xf8) == 0xf0) || ((b0 & 0xfc) == 0xf8) || ((b0 & 0xfe) == 0xfc)) {
int utfCount = 1;
if ((b0 & 0xf0) == 0xe0) utfCount = 2;
else if ((b0 & 0xf8) == 0xf0) utfCount = 3;
else if ((b0 & 0xfc) == 0xf8) utfCount = 4;
else if ((b0 & 0xfe) == 0xfc) utfCount = 5;
// 110xxxxx (10xxxxxx)+
// Range: U-00000080 - U-000007FF (count == 1)
// Range: U-00000800 - U-0000FFFF (count == 2)
// Range: U-00010000 - U-001FFFFF (count == 3)
// Range: U-00200000 - U-03FFFFFF (count == 4)
// Range: U-04000000 - U-7FFFFFFF (count == 5)
if (idx + utfCount > last) {
v[s++] = REPLACEMENT_CHAR;
continue;
}
// Extract usable bits from b0
int val = b0 & (0x1f >> (utfCount - 1));
for (int i = 0; i < utfCount; ++i) {
byte b = data[idx++];
if ((b & 0xc0) != 0x80) {
v[s++] = REPLACEMENT_CHAR;
idx--; // Put the input char back
continue outer;
}
// Push new bits in from the right side
val <<= 6;
val |= b & 0x3f;
}
// Note: Java allows overlong char
// specifications To disallow, check that val
// is greater than or equal to the minimum
// value for each count:
//
// count min value
// ----- ----------
// 1 0x80
// 2 0x800
// 3 0x10000
// 4 0x200000
// 5 0x4000000
// Allow surrogate values (0xD800 - 0xDFFF) to
// be specified using 3-byte UTF values only
if ((utfCount != 2) && (val >= 0xD800) && (val <= 0xDFFF)) {
v[s++] = REPLACEMENT_CHAR;
continue;
}
// Reject chars greater than the Unicode maximum of U+10FFFF.
if (val > 0x10FFFF) {
v[s++] = REPLACEMENT_CHAR;
continue;
}
// Encode chars from U+10000 up as surrogate pairs
if (val < 0x10000) {
v[s++] = (char) val;
} else {
int x = val & 0xffff;
int u = (val >> 16) & 0x1f;
int w = (u - 1) & 0xffff;
int hi = 0xd800 | (w << 6) | (x >> 10);
int lo = 0xdc00 | (x & 0x3ff);
v[s++] = (char) hi;
v[s++] = (char) lo;
}
} else {
// Illegal values 0x8*, 0x9*, 0xa*, 0xb*, 0xfd-0xff
v[s++] = REPLACEMENT_CHAR;
}
}
if (s == byteCount) {
// We guessed right, so we can use our temporary array as-is.
value = v;
} else {
// Our temporary array was too big, so reallocate and copy.
value = new char[s];
System.arraycopy(v, 0, value, 0, s);
}
return value;
}
我通过创建一个 20 字节的随机数组对此进行了单元测试,并与之前 Android 版本的原始 new String(bytes).toCharArray()
实现进行了比较,然后重复了一百万次。我没有看到多个旧 Android 版本有任何差异。
原始源代码来自这里:https://android.googlesource.com/platform/libcore/+/a7752f4d22097346dd7849b92b9f36d0a0a7a8f3/libdvm/src/main/java/java/lang/String.java#245
为了简化它,我删除了处理非UTF8字符集的部分,如果你使用new String()
,那么UTF-8是使用的默认字符集,所以你应该没问题。
此代码将使您获得所需的向后兼容性。但是正如其他人会建议你的那样,我建议尽可能寻找一个更简单的解决方案,它不依赖于 Android 版本(或其他不受你控制的组件)