SQLiteBlobTooBigException 在将请求分成 1 MB 的块后仍然发生（和 cursor.close()）
SQLiteBlobTooBigException still occurs after dividing the request in chunks of 1 MB (and cursor.close())
我的数据库可以直接导入一个6MB的文本文件。但是，无法提取文本，因为 CursorWindow 有 2MB 的限制。 （我应该使用文件，但有些用户已经遇到这个问题，我需要阅读整个文本才能将其放入文件中）
我使用 substr（一个特殊的 SQL 函数）只请求 1 MB 并且它起作用了。但是，下面的 while 循环在第二次迭代后不起作用（这意味着即使我调用 cursor.close()，CursorWindow 也没有被清空，所以对于第一次迭代它只有 1MB，但是在其次它有 2MB 并且抛出异常 SQLiteBlobTooBigException):
//Load in chunks
BookDbHelper bookDbHelper = new BookDbHelper(GlobalContext.get());
SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
//Query length
int chunk_size = (int) Math.pow(2, 20);//mb
String query_length = "SELECT _id, length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=" + id;
Cursor cursor_length = readableDatabase.rawQuery(query_length, null);
cursor_length.moveToFirst();
int length = cursor_length.getInt(1);
cursor_length.close();
bookDbHelper.close();
readableDatabase.close();
//Query text
int numSteps = length / chunk_size + 1;
int i = 0;
while(i < numSteps) {
BookDbHelper bookDbHelper2 = new BookDbHelper(GlobalContext.get());
SQLiteDatabase readableDatabase2 = bookDbHelper2.getReadableDatabase();
int from = i * chunk_size + 1;
int to = (i + 1) * chunk_size + 1;
//L.v(from + ", " + to);
String query = "SELECT _id, substr(text," + from + "," + to + ") FROM " + BookContract.TABLE_NAME + " WHERE _id=" + id;
Cursor cursor = readableDatabase2.rawQuery(query, null);
//Read
cursor.moveToFirst();
String string = cursor.getString(1);
cursor.close();
bookDbHelper2.close();
readableDatabase2.close();
//stringBuilder.append(string);
i++;
}
相关的列是 _id 和 text（其中包含一个非常大的字符串），相关的 sql 函数是 length()（知道必要的迭代次数）和 substr()（这样SQLiteBlobTooBigException 不会立即发生，因为未达到 2MB 限制）。
我尝试关闭 bookDbHelper 和 readableDatabase 但没有帮助。
如何强制关闭CursorWindow，让我发出1MB的请求，清空CursorWinow，然后继续发出其他请求？
How can I force CursorWindow to close so that I make a request of 1MB, empty the CursorWinow, and continue to make other requests?
我不认为关闭 Cursor 是你的问题，就好像不关闭 Cursor 会附加到 Cursor 并展开一样。
您的问题出在您构建的查询上。
简而言之，substr 函数不是 from to，它是 from for（for 是返回字符串的 size/length。您的计算基于第 2 个值是字符的偏移量）。因此，提取的字符串的长度随着块大小的增加而增加，直到它在减少时超过字符串的末尾（在此之前吹掉 CursorWindow）。
因此使用 1MB 的第二个块（如果被视为使用偏移量）在第二次 运行 注定要失败，因为它实际上是要提取的长度 (2MB)。减少到小于 1MB 将允许一些余地，但可能会破坏 CursorWindow（但会获得额外的数据）。
但是，作为替代方案，使用单个游标将每个块作为一个提取的行。解决方案可能是：-
//Load in chunks
BookDbHelper bookDbHelper = new BookDbHelper(/*GlobalContext.get()*/this);
SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
//Query length
StringBuilder wholeBookText = new StringBuilder();
int chunk_size = (int) Math.pow(2, 20);//mb
String query_length = "SELECT length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
Cursor cursor = readableDatabase.rawQuery(query_length, new String[]{String.valueOf(id)});
int length = 0;
if (cursor.moveToFirst()) {
length = cursor.getInt(0);
}
int numSteps = length / chunk_size + 1;
int i = 0;
Log.d("BOOKINFO", "Length of Text is " + length + " Number of Chunks = " + numSteps + " Chunk Size = " + chunk_size);
StringBuilder sb = new StringBuilder();
for (i=1; i < length + 1; i+= chunk_size) {
if (sb.length() > 1) sb.append(" UNION ALL ");
sb.append("SELECT substr(text,")
.append(String.valueOf(i)).append(",").append(String.valueOf(chunk_size))
.append(") FROM ").append(BookContract.TABLE_NAME)
.append(" WHERE _id=").append(String.valueOf(id));
}
sb.append(";");
Log.d("BOOKINFOV2","SQL generated :-\n\t" + sb.toString());
cursor = readableDatabase.rawQuery(sb.toString(),null);
wholeBookText = new StringBuilder();
while (cursor.moveToNext()) {
wholeBookText.append(cursor.getString(0));
Log.d("BOOKINFO","Obtained String who's length is " + cursor.getString(0).length() + "\n\tTotal Extracted = " + wholeBookText.length());
}
而不是循环中的个别查询 运行。这会生成一个查询，将每个块提取为一行。也就是说，它在所有查询之间建立了一个联合。例如
SELECT substr(text,1,1048576) FROM book WHERE _id=4
UNION ALL SELECT substr(text,1048577,1048576) FROM book WHERE _id=4
UNION ALL SELECT substr(text,2097153,1048576) FROM book WHERE _id=4
UNION ALL SELECT substr(text,3145729,1048576) FROM book WHERE _id=4;
- 取自上述测试运行。
- 可以看出到（应该是for）是chunk的大小。最后一个块将根据剩余数据进行分类运行。
测试的完整输出 运行 :-
2019-12-16 14:21:35.546 D/BOOKINFOV2: SQL generated :-
SELECT substr(text,1,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,1048577,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,2097153,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,3145729,1048576) FROM book WHERE _id=4;
2019-12-16 14:21:35.555 W/CursorWindow: Window is full: requested allocation 1048577 bytes, free space 1048128 bytes, window size 2097152 bytes
2019-12-16 14:21:35.585 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 1048576
2019-12-16 14:21:35.599 W/CursorWindow: Window is full: requested allocation 1048577 bytes, free space 1048128 bytes, window size 2097152 bytes
2019-12-16 14:21:35.616 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 2097152
2019-12-16 14:21:35.653 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 3145728
2019-12-16 14:21:35.654 D/BOOKINFO: Obtained String who's length is 51
Total Extracted = 3145779
- 如您所见，CursorWindow 会溢出，但该行未添加，下次添加并可访问。
当然，您可以采用多查询方法，在这种情况下，代码可以是：-
//Load in chunks
BookDbHelper bookDbHelper = new BookDbHelper(/*GlobalContext.get()*/this);
SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
//Query length
StringBuilder wholeBookText = new StringBuilder();
int chunk_size = (int) Math.pow(2, 19);//mb
chunk_size = (1024 * 1024);
String query_length = "SELECT length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
Cursor cursor = readableDatabase.rawQuery(query_length, new String[]{String.valueOf(id)});
int length = 0;
if (cursor.moveToFirst()) {
length = cursor.getInt(0);
}
int numSteps = length / chunk_size + 1;
int i = 0;
Log.d("BOOKINFO", "Length of Text is " + length + " Number of Chunks = " + numSteps + " Chunk Size = " + chunk_size);
int from = 1, to = chunk_size;
while (i < numSteps && length > 0) {
if (to > length) to = length;
String query = "SELECT substr(text," + from + "," + (chunk_size) + ") FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
Log.d("BOOKINFOSQL",query);
cursor.close();
cursor = readableDatabase.rawQuery(query, new String[]{String.valueOf(id)});
//Read
if (cursor.moveToFirst()) {
wholeBookText.append(cursor.getString(0));
Log.d("BOOKINFO","Obtained String who's length is " + cursor.getString(0).length() + "\n\tTotal Extracted = " + wholeBookText.length());
}
cursor.close();
i++;
from = (i * chunk_size) + 1;
to = from + chunk_size;
}
if (!cursor.isClosed()) {
cursor.close();
}
Log.d("BOOKINFO", "The length of the extracted data is " + wholeBookText.length());
以上结果：-
2019-12-16 14:16:15.336 D/BOOKINFO: Length of Text is 3145779 Number of Chunks = 4 Chunk Size = 1048576
2019-12-16 14:16:15.336 D/BOOKINFOSQL: SELECT substr(text,1,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.358 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 1048576
2019-12-16 14:16:15.358 D/BOOKINFOSQL: SELECT substr(text,1048577,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.382 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 2097152
2019-12-16 14:16:15.383 D/BOOKINFOSQL: SELECT substr(text,2097153,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.409 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 3145728
2019-12-16 14:16:15.409 D/BOOKINFOSQL: SELECT substr(text,3145729,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.418 D/BOOKINFO: Obtained String who's length is 51
Total Extracted = 3145779
2019-12-16 14:16:15.418 D/BOOKINFO: The length of the extracted data is 3145779
我的数据库可以直接导入一个6MB的文本文件。但是，无法提取文本，因为 CursorWindow 有 2MB 的限制。 （我应该使用文件，但有些用户已经遇到这个问题，我需要阅读整个文本才能将其放入文件中） 我使用 substr（一个特殊的 SQL 函数）只请求 1 MB 并且它起作用了。但是，下面的 while 循环在第二次迭代后不起作用（这意味着即使我调用 cursor.close()，CursorWindow 也没有被清空，所以对于第一次迭代它只有 1MB，但是在其次它有 2MB 并且抛出异常 SQLiteBlobTooBigException):
//Load in chunks
BookDbHelper bookDbHelper = new BookDbHelper(GlobalContext.get());
SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
//Query length
int chunk_size = (int) Math.pow(2, 20);//mb
String query_length = "SELECT _id, length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=" + id;
Cursor cursor_length = readableDatabase.rawQuery(query_length, null);
cursor_length.moveToFirst();
int length = cursor_length.getInt(1);
cursor_length.close();
bookDbHelper.close();
readableDatabase.close();
//Query text
int numSteps = length / chunk_size + 1;
int i = 0;
while(i < numSteps) {
BookDbHelper bookDbHelper2 = new BookDbHelper(GlobalContext.get());
SQLiteDatabase readableDatabase2 = bookDbHelper2.getReadableDatabase();
int from = i * chunk_size + 1;
int to = (i + 1) * chunk_size + 1;
//L.v(from + ", " + to);
String query = "SELECT _id, substr(text," + from + "," + to + ") FROM " + BookContract.TABLE_NAME + " WHERE _id=" + id;
Cursor cursor = readableDatabase2.rawQuery(query, null);
//Read
cursor.moveToFirst();
String string = cursor.getString(1);
cursor.close();
bookDbHelper2.close();
readableDatabase2.close();
//stringBuilder.append(string);
i++;
}
相关的列是 _id 和 text（其中包含一个非常大的字符串），相关的 sql 函数是 length()（知道必要的迭代次数）和 substr()（这样SQLiteBlobTooBigException 不会立即发生，因为未达到 2MB 限制）。
我尝试关闭 bookDbHelper 和 readableDatabase 但没有帮助。
如何强制关闭CursorWindow，让我发出1MB的请求，清空CursorWinow，然后继续发出其他请求？
How can I force CursorWindow to close so that I make a request of 1MB, empty the CursorWinow, and continue to make other requests?
我不认为关闭 Cursor 是你的问题，就好像不关闭 Cursor 会附加到 Cursor 并展开一样。
您的问题出在您构建的查询上。
简而言之，substr 函数不是 from to，它是 from for（for 是返回字符串的 size/length。您的计算基于第 2 个值是字符的偏移量）。因此，提取的字符串的长度随着块大小的增加而增加，直到它在减少时超过字符串的末尾（在此之前吹掉 CursorWindow）。
因此使用 1MB 的第二个块（如果被视为使用偏移量）在第二次 运行 注定要失败，因为它实际上是要提取的长度 (2MB)。减少到小于 1MB 将允许一些余地，但可能会破坏 CursorWindow（但会获得额外的数据）。
但是，作为替代方案，使用单个游标将每个块作为一个提取的行。解决方案可能是：-
//Load in chunks
BookDbHelper bookDbHelper = new BookDbHelper(/*GlobalContext.get()*/this);
SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
//Query length
StringBuilder wholeBookText = new StringBuilder();
int chunk_size = (int) Math.pow(2, 20);//mb
String query_length = "SELECT length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
Cursor cursor = readableDatabase.rawQuery(query_length, new String[]{String.valueOf(id)});
int length = 0;
if (cursor.moveToFirst()) {
length = cursor.getInt(0);
}
int numSteps = length / chunk_size + 1;
int i = 0;
Log.d("BOOKINFO", "Length of Text is " + length + " Number of Chunks = " + numSteps + " Chunk Size = " + chunk_size);
StringBuilder sb = new StringBuilder();
for (i=1; i < length + 1; i+= chunk_size) {
if (sb.length() > 1) sb.append(" UNION ALL ");
sb.append("SELECT substr(text,")
.append(String.valueOf(i)).append(",").append(String.valueOf(chunk_size))
.append(") FROM ").append(BookContract.TABLE_NAME)
.append(" WHERE _id=").append(String.valueOf(id));
}
sb.append(";");
Log.d("BOOKINFOV2","SQL generated :-\n\t" + sb.toString());
cursor = readableDatabase.rawQuery(sb.toString(),null);
wholeBookText = new StringBuilder();
while (cursor.moveToNext()) {
wholeBookText.append(cursor.getString(0));
Log.d("BOOKINFO","Obtained String who's length is " + cursor.getString(0).length() + "\n\tTotal Extracted = " + wholeBookText.length());
}
而不是循环中的个别查询 运行。这会生成一个查询，将每个块提取为一行。也就是说，它在所有查询之间建立了一个联合。例如
SELECT substr(text,1,1048576) FROM book WHERE _id=4
UNION ALL SELECT substr(text,1048577,1048576) FROM book WHERE _id=4
UNION ALL SELECT substr(text,2097153,1048576) FROM book WHERE _id=4
UNION ALL SELECT substr(text,3145729,1048576) FROM book WHERE _id=4;
- 取自上述测试运行。
- 可以看出到（应该是for）是chunk的大小。最后一个块将根据剩余数据进行分类运行。
测试的完整输出 运行 :-
2019-12-16 14:21:35.546 D/BOOKINFOV2: SQL generated :-
SELECT substr(text,1,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,1048577,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,2097153,1048576) FROM book WHERE _id=4 UNION ALL SELECT substr(text,3145729,1048576) FROM book WHERE _id=4;
2019-12-16 14:21:35.555 W/CursorWindow: Window is full: requested allocation 1048577 bytes, free space 1048128 bytes, window size 2097152 bytes
2019-12-16 14:21:35.585 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 1048576
2019-12-16 14:21:35.599 W/CursorWindow: Window is full: requested allocation 1048577 bytes, free space 1048128 bytes, window size 2097152 bytes
2019-12-16 14:21:35.616 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 2097152
2019-12-16 14:21:35.653 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 3145728
2019-12-16 14:21:35.654 D/BOOKINFO: Obtained String who's length is 51
Total Extracted = 3145779
- 如您所见，CursorWindow 会溢出，但该行未添加，下次添加并可访问。
当然，您可以采用多查询方法，在这种情况下，代码可以是：-
//Load in chunks
BookDbHelper bookDbHelper = new BookDbHelper(/*GlobalContext.get()*/this);
SQLiteDatabase readableDatabase = bookDbHelper.getReadableDatabase();
//Query length
StringBuilder wholeBookText = new StringBuilder();
int chunk_size = (int) Math.pow(2, 19);//mb
chunk_size = (1024 * 1024);
String query_length = "SELECT length(text) FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
Cursor cursor = readableDatabase.rawQuery(query_length, new String[]{String.valueOf(id)});
int length = 0;
if (cursor.moveToFirst()) {
length = cursor.getInt(0);
}
int numSteps = length / chunk_size + 1;
int i = 0;
Log.d("BOOKINFO", "Length of Text is " + length + " Number of Chunks = " + numSteps + " Chunk Size = " + chunk_size);
int from = 1, to = chunk_size;
while (i < numSteps && length > 0) {
if (to > length) to = length;
String query = "SELECT substr(text," + from + "," + (chunk_size) + ") FROM " + BookContract.TABLE_NAME + " WHERE _id=?";
Log.d("BOOKINFOSQL",query);
cursor.close();
cursor = readableDatabase.rawQuery(query, new String[]{String.valueOf(id)});
//Read
if (cursor.moveToFirst()) {
wholeBookText.append(cursor.getString(0));
Log.d("BOOKINFO","Obtained String who's length is " + cursor.getString(0).length() + "\n\tTotal Extracted = " + wholeBookText.length());
}
cursor.close();
i++;
from = (i * chunk_size) + 1;
to = from + chunk_size;
}
if (!cursor.isClosed()) {
cursor.close();
}
Log.d("BOOKINFO", "The length of the extracted data is " + wholeBookText.length());
以上结果：-
2019-12-16 14:16:15.336 D/BOOKINFO: Length of Text is 3145779 Number of Chunks = 4 Chunk Size = 1048576
2019-12-16 14:16:15.336 D/BOOKINFOSQL: SELECT substr(text,1,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.358 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 1048576
2019-12-16 14:16:15.358 D/BOOKINFOSQL: SELECT substr(text,1048577,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.382 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 2097152
2019-12-16 14:16:15.383 D/BOOKINFOSQL: SELECT substr(text,2097153,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.409 D/BOOKINFO: Obtained String who's length is 1048576
Total Extracted = 3145728
2019-12-16 14:16:15.409 D/BOOKINFOSQL: SELECT substr(text,3145729,1048576) FROM book WHERE _id=?
2019-12-16 14:16:15.418 D/BOOKINFO: Obtained String who's length is 51
Total Extracted = 3145779
2019-12-16 14:16:15.418 D/BOOKINFO: The length of the extracted data is 3145779