为什么我所有的解码字符串都有“？”在末尾？ Java 字符串解码
Why do all my decoded Strings have '?' at the end? Java String decoding
我正在使用 Tweepy 库 (Python) 和 Kafka 从 Twitter 检索推文。文本以 UTF-8 编码，如下行所示：
self.producer.send('my-topic', data.encode('UTF-8'))
其中 "data" 是一个字符串。然后，此数据以键值格式存储到 Oracle NoSQL 数据库中。出于这个原因，推文本身是经过编码的。我这样做 Java:
Value myValue = Value.createValue(msg.value().getBytes("UTF-8"));
最后，推文由 Java 开发的 Formatter 检索。为了将其存储在关系模式中，我必须解析推文，以便将其作为字符串检索。
String data = new String(value.toByteArray(),StandardCharsets.UTF_8);
如您所见，我在所有步骤中都使用 UTF-8 编码。但是，当我在我的数据库中看到推文的文本时，它总是被删减。例如：
RT @briIIohead: the hardest pill i had to swallow this year was learning that no matter how good you could be to somebody, no matter how mu?
注意它是如何以“?”结尾的符号，而且已经被清楚地切断了。好吧，每条长推文都会发生这种情况。我的意思是，如果文本长度为 30 个字符，那么它显示正常，但是任何超过 100 个字符的内容都会被剪切。
起初我以为这可能是我的table定义，但是字段"Text"被声明为VARCHAR2(400 CHAR)这是推文在社交中可以拥有的最大字符数网络。
关于如何发现剪切文本和放置“？”的任何想法符号在最后？
"data" 的样子：
{"created_at":"Tue May 28 09:23:36 +0000 2019","id":1133302792129351681,"id_str":"1133302792129351681","text":"RT @AppleEDU: Learn, create, and do more with iPad in your classroom. Get the free Everyone Can Create curriculum and bring projects to lif\u2026","source":"\u003ca href=\"http:\/\/twitter.com\/download\/iphone\" rel=\"nofollow\"\u003eTwitter for iPhone\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":1060510851889750022,"id_str":"1060510851889750022","name":"Rem.0112","screen_name":"0112Rem","location":"Mawson Lakes, Adelaide","url":null,"description":null,"translator_type":"none","protected":false,"verified":false,"followers_count":739,"friends_count":1853,"listed_count":10,"favourites_count":33406,"statuses_count":36936,"created_at":"Thu Nov 08 12:34:25 +0000 2018","utc_offset":null,"time_zone":null,"geo_enabled":true,"lang":null,"contributors_enabled":false,"is_translator":false,"profile_background_color":"F5F8FA","profile_background_image_url":"","profile_background_image_url_https":"","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/1093157842163355649\/6oAdJTCs_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/1093157842163355649\/6oAdJTCs_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/1060510851889750022\/1546155144","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweeted_status":{"created_at":"Thu May 23 15:15:16 +0000 2019","id":1131579354964725760,"id_str":"1131579354964725760","text":"Learn, create, and do more with iPad in your classroom. Get the free Everyone Can Create curriculum and bring proje\u2026 https:\/\/t.co\/aeeSPTXtFx","source":"\u003ca href=\"https:\/\/ads-api.twitter.com\" rel=\"nofollow\"\u003eTwitter Ads Composer\u003c\/a\u003e","truncated":true,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":468741166,"id_str":"468741166","name":"Apple Education","screen_name":"AppleEDU","location":"Cupertino, CA","url":null,"description":"Spark new ideas, create more aha moments, and teach in ways you\u2019ve always imagined. Follow @AppleEDU for tips, updates, and inspiration.","translator_type":"none","protected":false,"verified":true,"followers_count":728781,"friends_count":273,"listed_count":2594,"favourites_count":13189,"statuses_count":2766,"created_at":"Thu Jan 19 21:26:14 +0000 2012","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":null,"contributors_enabled":false,"is_translator":false,"profile_background_color":"F0F0F0","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_tile":false,"profile_link_color":"0088CC","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":false,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/892429342046691328\/2SOlm_09_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/892429342046691328\/2SOlm_09_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/468741166\/1530123538","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"extended_tweet":{"full_text":"Learn, create, and do more with iPad in your classroom. Get the free Everyone Can Create curriculum and bring projects to life through music, drawing, video and photography.","display_text_range":[0,173],"entities":{"hashtags":[],"urls":[],"user_mentions":[],"symbols":[]}},"quote_count":0,"reply_count":3,"retweet_count":3,"favorite_count":58,"entities":{"hashtags":[],"urls":[{"url":"https:\/\/t.co\/aeeSPTXtFx","expanded_url":"https:\/\/twitter.com\/i\/web\/status\/1131579354964725760","display_url":"twitter.com\/i\/web\/status\/1\u2026","indices":[117,140]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"scopes":{"followers":false},"filter_level":"low","lang":"en"},"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"urls":[],"user_mentions":[{"screen_name":"AppleEDU","name":"Apple Education","id":468741166,"id_str":"468741166","indices":[3,12]}],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"en","timestamp_ms":"1559035416048"}
我还必须提到，这整个块都是经过编码的。然后解码，最后解析到数据库中。所有字段都被正确解码和解析，除了 "text" 被剪切
根据 official 文档，推文不超过“140”个字符（这是一个宽泛的定义）；但最近他们将其更改为 280。
同一份文件说：
Twitter counts the length of a Tweet using the Normalization Form C (NFC) version of the text.
所以他们首先规范化文本（我会让你弄清楚这是如何完成的 java）。后来他们说：
Twitter also counts the number of codepoints in the text rather than UTF-8 bytes.
因此：
String test = "RT @briIIohead: the hardest pill i had to swallow this year was learning that no matter how good you could be to somebody, no matter how mu";
System.out.println(test.codePoints().count()); // 139
最初的推文似乎是 280 "characters" 而您使用的库并不知道这一点，因此它只使用了之前的 140 个。由于这会进行一些分块，因此分块似乎也是错误的，它在末尾删除了一些 "partial" 字节。当你尝试打印那些 - java 不知道那些（最后）字节的实际含义（因为一些错误的分块）并且只是说 ? （这是显示内容的默认策略当它根本不理解某些东西时）。
我正在使用 Tweepy 库 (Python) 和 Kafka 从 Twitter 检索推文。文本以 UTF-8 编码，如下行所示：
self.producer.send('my-topic', data.encode('UTF-8'))
其中 "data" 是一个字符串。然后，此数据以键值格式存储到 Oracle NoSQL 数据库中。出于这个原因，推文本身是经过编码的。我这样做 Java:
Value myValue = Value.createValue(msg.value().getBytes("UTF-8"));
最后，推文由 Java 开发的 Formatter 检索。为了将其存储在关系模式中，我必须解析推文，以便将其作为字符串检索。
String data = new String(value.toByteArray(),StandardCharsets.UTF_8);
如您所见，我在所有步骤中都使用 UTF-8 编码。但是，当我在我的数据库中看到推文的文本时，它总是被删减。例如：
RT @briIIohead: the hardest pill i had to swallow this year was learning that no matter how good you could be to somebody, no matter how mu?
注意它是如何以“?”结尾的符号，而且已经被清楚地切断了。好吧，每条长推文都会发生这种情况。我的意思是，如果文本长度为 30 个字符，那么它显示正常，但是任何超过 100 个字符的内容都会被剪切。
起初我以为这可能是我的table定义，但是字段"Text"被声明为VARCHAR2(400 CHAR)这是推文在社交中可以拥有的最大字符数网络。
关于如何发现剪切文本和放置“？”的任何想法符号在最后？
"data" 的样子：
{"created_at":"Tue May 28 09:23:36 +0000 2019","id":1133302792129351681,"id_str":"1133302792129351681","text":"RT @AppleEDU: Learn, create, and do more with iPad in your classroom. Get the free Everyone Can Create curriculum and bring projects to lif\u2026","source":"\u003ca href=\"http:\/\/twitter.com\/download\/iphone\" rel=\"nofollow\"\u003eTwitter for iPhone\u003c\/a\u003e","truncated":false,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":1060510851889750022,"id_str":"1060510851889750022","name":"Rem.0112","screen_name":"0112Rem","location":"Mawson Lakes, Adelaide","url":null,"description":null,"translator_type":"none","protected":false,"verified":false,"followers_count":739,"friends_count":1853,"listed_count":10,"favourites_count":33406,"statuses_count":36936,"created_at":"Thu Nov 08 12:34:25 +0000 2018","utc_offset":null,"time_zone":null,"geo_enabled":true,"lang":null,"contributors_enabled":false,"is_translator":false,"profile_background_color":"F5F8FA","profile_background_image_url":"","profile_background_image_url_https":"","profile_background_tile":false,"profile_link_color":"1DA1F2","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":true,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/1093157842163355649\/6oAdJTCs_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/1093157842163355649\/6oAdJTCs_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/1060510851889750022\/1546155144","default_profile":true,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"retweeted_status":{"created_at":"Thu May 23 15:15:16 +0000 2019","id":1131579354964725760,"id_str":"1131579354964725760","text":"Learn, create, and do more with iPad in your classroom. Get the free Everyone Can Create curriculum and bring proje\u2026 https:\/\/t.co\/aeeSPTXtFx","source":"\u003ca href=\"https:\/\/ads-api.twitter.com\" rel=\"nofollow\"\u003eTwitter Ads Composer\u003c\/a\u003e","truncated":true,"in_reply_to_status_id":null,"in_reply_to_status_id_str":null,"in_reply_to_user_id":null,"in_reply_to_user_id_str":null,"in_reply_to_screen_name":null,"user":{"id":468741166,"id_str":"468741166","name":"Apple Education","screen_name":"AppleEDU","location":"Cupertino, CA","url":null,"description":"Spark new ideas, create more aha moments, and teach in ways you\u2019ve always imagined. Follow @AppleEDU for tips, updates, and inspiration.","translator_type":"none","protected":false,"verified":true,"followers_count":728781,"friends_count":273,"listed_count":2594,"favourites_count":13189,"statuses_count":2766,"created_at":"Thu Jan 19 21:26:14 +0000 2012","utc_offset":null,"time_zone":null,"geo_enabled":false,"lang":null,"contributors_enabled":false,"is_translator":false,"profile_background_color":"F0F0F0","profile_background_image_url":"http:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_image_url_https":"https:\/\/abs.twimg.com\/images\/themes\/theme1\/bg.png","profile_background_tile":false,"profile_link_color":"0088CC","profile_sidebar_border_color":"C0DEED","profile_sidebar_fill_color":"DDEEF6","profile_text_color":"333333","profile_use_background_image":false,"profile_image_url":"http:\/\/pbs.twimg.com\/profile_images\/892429342046691328\/2SOlm_09_normal.jpg","profile_image_url_https":"https:\/\/pbs.twimg.com\/profile_images\/892429342046691328\/2SOlm_09_normal.jpg","profile_banner_url":"https:\/\/pbs.twimg.com\/profile_banners\/468741166\/1530123538","default_profile":false,"default_profile_image":false,"following":null,"follow_request_sent":null,"notifications":null},"geo":null,"coordinates":null,"place":null,"contributors":null,"is_quote_status":false,"extended_tweet":{"full_text":"Learn, create, and do more with iPad in your classroom. Get the free Everyone Can Create curriculum and bring projects to life through music, drawing, video and photography.","display_text_range":[0,173],"entities":{"hashtags":[],"urls":[],"user_mentions":[],"symbols":[]}},"quote_count":0,"reply_count":3,"retweet_count":3,"favorite_count":58,"entities":{"hashtags":[],"urls":[{"url":"https:\/\/t.co\/aeeSPTXtFx","expanded_url":"https:\/\/twitter.com\/i\/web\/status\/1131579354964725760","display_url":"twitter.com\/i\/web\/status\/1\u2026","indices":[117,140]}],"user_mentions":[],"symbols":[]},"favorited":false,"retweeted":false,"scopes":{"followers":false},"filter_level":"low","lang":"en"},"is_quote_status":false,"quote_count":0,"reply_count":0,"retweet_count":0,"favorite_count":0,"entities":{"hashtags":[],"urls":[],"user_mentions":[{"screen_name":"AppleEDU","name":"Apple Education","id":468741166,"id_str":"468741166","indices":[3,12]}],"symbols":[]},"favorited":false,"retweeted":false,"filter_level":"low","lang":"en","timestamp_ms":"1559035416048"}
我还必须提到，这整个块都是经过编码的。然后解码，最后解析到数据库中。所有字段都被正确解码和解析，除了 "text" 被剪切
根据 official 文档，推文不超过“140”个字符（这是一个宽泛的定义）；但最近他们将其更改为 280。
同一份文件说：
Twitter counts the length of a Tweet using the Normalization Form C (NFC) version of the text.
所以他们首先规范化文本（我会让你弄清楚这是如何完成的 java）。后来他们说：
Twitter also counts the number of codepoints in the text rather than UTF-8 bytes.
因此：
String test = "RT @briIIohead: the hardest pill i had to swallow this year was learning that no matter how good you could be to somebody, no matter how mu";
System.out.println(test.codePoints().count()); // 139
最初的推文似乎是 280 "characters" 而您使用的库并不知道这一点，因此它只使用了之前的 140 个。由于这会进行一些分块，因此分块似乎也是错误的，它在末尾删除了一些 "partial" 字节。当你尝试打印那些 - java 不知道那些（最后）字节的实际含义（因为一些错误的分块）并且只是说 ? （这是显示内容的默认策略当它根本不理解某些东西时）。