C Python API Extensions 忽略 open(errors="ignore") 并继续抛出编码异常

Question

给定一个包含无效 UTF8 的文件 /myfiles/file_with_invalid_encoding.txt：

parse this correctly
FÃ¸Ã¶»BÃ¥r
also parse this correctly

我正在使用 C API 中的内置 Python open 函数，如下最小示例（不包括 C Python 设置样板）：

const char* filepath = "/myfiles/file_with_invalid_encoding.txt";
PyObject* iomodule = PyImport_ImportModule( "builtins" );

if( iomodule == NULL ) {
    PyErr_PrintEx(100); return;
}
PyObject* openfunction = PyObject_GetAttrString( iomodule, "open" );

if( openfunction == NULL ) {
    PyErr_PrintEx(100); return;
}

PyObject* openfile = PyObject_CallFunction( openfunction, 
       "s", filepath, "s", "r", "i", -1, "s", "UTF8", "s", "ignore" );

if( openfile == NULL ) {
    PyErr_PrintEx(100); return;
}

PyObject* iterfunction = PyObject_GetAttrString( openfile, "__iter__" );
Py_DECREF( openfunction );

if( iterfunction == NULL ) {
    PyErr_PrintEx(100); return;
}
PyObject* openfileresult = PyObject_CallObject( iterfunction, NULL );
Py_DECREF( iterfunction );

if( openfileresult == NULL ) {
    PyErr_PrintEx(100); return;
}
PyObject* fileiterator = PyObject_GetAttrString( openfile, "__next__" );
Py_DECREF( openfileresult );

if( fileiterator == NULL ) {
    PyErr_PrintEx(100); return;
}
PyObject* readline;
std::cout << "Here 1!" << std::endl;

while( ( readline = PyObject_CallObject( fileiterator, NULL ) ) != NULL ) {
    std::cout << "Here 2!" << std::endl;
    std::cout << PyUnicode_AsUTF8( readline ) << std::endl;
    Py_DECREF( readline );
}
PyErr_PrintEx(100);
PyErr_Clear();

PyObject* closefunction = PyObject_GetAttrString( openfile, "close" );

if( closefunction == NULL ) {
    PyErr_PrintEx(100); return;
}

PyObject* closefileresult = PyObject_CallObject( closefunction, NULL );
Py_DECREF( closefunction );

if( closefileresult == NULL ) {
    PyErr_PrintEx(100); return;
}

Py_XDECREF( closefileresult );
Py_XDECREF( iomodule );
Py_XDECREF( openfile );
Py_XDECREF( fileiterator );

我正在调用传递 ignore 参数的 open 函数来忽略编码错误，但是 Python 忽略我并在发现无效的 UTF8 字符时不断抛出编码异常：

Here 1!
Traceback (most recent call last):
  File "/usr/lib/python3.6/codecs.py", line 321, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xbb in position 26: invalid start byte

正如您在上面和下面看到的，当我调用 builtins.open() 函数时，我传递了 ignore 参数，但它没有任何效果。我也尝试将 ignore 更改为 replace，但 C Python 无论如何都会抛出异常：

PyObject* openfile = PyObject_CallFunction( openfunction, 
       "s", filepath, "s", "r", "i", -1, "s", "UTF8", "s", "ignore" );

Answer 1

我设法通过替换函数 PyObject_CallFunction with PyObject_CallFunctionObjArgs 函数来修复它：

PyObject* openfile = PyObject_CallFunction( openfunction, 
       "s", filepath, "s", "r", "i", -1, "s", "UTF8", "s", "ignore" );
// -->
PyObject* filepathpy = Py_BuildValue( "s", filepath );
PyObject* openmodepy = Py_BuildValue( "s", "r" );
PyObject* buffersizepy = Py_BuildValue( "i", -1 );
PyObject* encodingpy = Py_BuildValue( "s", "UTF-8" );
PyObject* ignorepy = Py_BuildValue( "s", "ignore" );

PyObject* openfile = PyObject_CallFunctionObjArgs( openfunction, 
        filepathpy, openmodepy, buffersizepy, encodingpy, ignorepy, NULL );

C Python 喜欢的长版本：

PyObject* filepathpy = Py_BuildValue( "s", filepath );
if( filepathpy == NULL ) {
    PyErr_PrintEx(100); return;
}

PyObject* openmodepy = Py_BuildValue( "s", "r" );
if( openmodepy == NULL ) {
    PyErr_PrintEx(100); return;
}

PyObject* buffersizepy = Py_BuildValue( "i", -1 );
if( buffersizepy == NULL ) {
    PyErr_PrintEx(100); return;
}

PyObject* encodingpy = Py_BuildValue( "s", "UTF-8" );
if( encodingpy == NULL ) {
    PyErr_PrintEx(100); return;
}

PyObject* ignorepy = Py_BuildValue( "s", "ignore" );
if( ignorepy == NULL ) {
    PyErr_PrintEx(100); return;
}

PyObject* openfile = PyObject_CallFunctionObjArgs( openfunction,
        filepathpy, openmodepy, buffersizepy, encodingpy, ignorepy, NULL );

Py_DECREF( filepathpy );
Py_DECREF( openmodepy );
Py_DECREF( buffersizepy );
Py_DECREF( encodingpy );
Py_DECREF( ignorepy );

if( openfile == NULL ) {
    PyErr_PrintEx(100); return;
}

Answer 2

PyObject_CallFunction（以及 Py_BuildValue 和其他）采用描述所有参数的单一格式字符串。当你这样做时

PyObject* openfile = PyObject_CallFunction( openfunction, 
   "s", filepath, "s", "r", "i", -1, "s", "UTF8", "s", "ignore" );

你说了 "one string argument" 并且 filepath 之后的所有参数都被忽略了。相反，你应该这样做：

PyObject* openfile = PyObject_CallFunction( openfunction, 
   "ssiss", filepath, "r", -1, "UTF8", "ignore" );

说“5 个参数：2 个字符串和 int，以及另外两个字符串”。即使您选择使用其他 PyObject_Call* 函数之一，您也会发现以这种方式使用 Py_BuildValue 会更容易。

C Python API Extensions 忽略 open(errors="ignore") 并继续抛出编码异常

C Python API Extensions is ignoring open(errors="ignore") and keeps throwing the encoding exception anyways

c

python

cpython

python-c-api