如何extend/reusePythonCExtensions/API执行？
How to extend/reuse Python C Extensions/API implementation?
问题是现在，我必须使用 Posix C getline 函数从文件中获取行，然后才将其转换为 Python Unicode 对象使用PyUnicode_DecodeUTF8 并使用我的 caching policy algorithm. This process is 性能缓存它与 Python 内置 for line in file C 实现相比。
如果我从我的代码中删除 PyUnicode_DecodeUTF8 调用，那么，我使用 Posix C getline 的实现变得 5% 比 Python 更快内置 for line in file C 实现。所以，如果我可以让 Python 直接给我一个 Python Unicode String 对象，而不是必须先调用 Posix C getline 函数（然后才转换它的结果到 Python Unicode 对象），我的代码性能将提高几乎 20%（从最大值 23%），即它不会变成 100% 等同于 for line in file 性能，因为我通过缓存东西做了一些工作，但是这种开销很小。
例如，我想采用 _textiowrapper_readline() 函数并在我的代码中使用它，如下所示：
#include <Python.h>
#include <textio.c.h> // C Python file defininig:
// _textiowrapper_readline(),
// CHECK_ATTACHED(),
// PyUnicode_READY(), etc
typedef struct
{
PyObject_HEAD
}
PyMymoduleExtendingPython;
static PyObject*
PyMymoduleExtendingPython_iternext(PyMymoduleExtendingPython* self, PyObject* args)
{
PyObject *line;
CHECK_ATTACHED(self);
line = _textiowrapper_readline(self, -1); // <- function from `textio.c`
if (line == NULL || PyUnicode_READY(line) == -1)
return NULL;
if (PyUnicode_GET_LENGTH(line) == 0) {
/* Reached EOF or would have blocked */
Py_DECREF(line);
Py_CLEAR(self->snapshot);
self->telling = self->seekable;
return NULL;
}
return line;
}
// create my module
PyMODINIT_FUNC PyInit_mymodule_extending_python_api(void)
{
PyObject* mymodule;
PyMymoduleExtendingPython.tp_iternext =
(iternextfunc) PyMymoduleExtendingPython_iternext;
Py_INCREF( &PyMymoduleExtendingPython );
PyModule_AddObject( mymodule, "FastFile", (PyObject*) &PyMymoduleExtendingPython );
return mymodule;
}
我如何包含 C Python 中的 textio 实现并在我自己的 Python C Extension/API 中重用它的代码？
如我在上一个问题 中所述，用于读取行的 Python 内置方法比我自己使用 C 或 C++ 标准方法编写自己的方法从文件中获取行要快。
在 ，有人建议我通过读取 8KB 的块然后调用 PyUnicode_DecodeUTF8 来解码它们，而不是调用 [=16] 来重新实现 Python 算法=] 在我阅读的每一行。
但是，与其重写所有已经 written/done/ready 的 C Python 代码来读取行，我可以调用它的 "getline" 函数 _textiowrapper_readline() 直接获取该行一个 Python Unicode 对象，然后缓存 it/use 就像我已经在处理从 Posix C getline 函数中得到的行一样（并传递给 PyUnicode_DecodeUTF8() 解码它们变成 Python Unicode 对象）。
我没有设法直接导入 C API（扩展）函数，但我使用 Python 导入了 io 模块，它有一个 link/reference到全局内置函数 open 作为 io.open().
bool hasfinished;
const char* filepath;
long long int linecount;
std::deque<PyObject*> linecache;
PyObject* iomodule;
PyObject* openfile;
PyObject* fileiterator;
FastFile(const char* filepath) : hasfinished(false), filepath(filepath), linecount(0) {
iomodule = PyImport_ImportModule( "io" );
if( iomodule == NULL ) {
std::cerr << "ERROR: FastFile failed to import the io module '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
PyObject* openfunction = PyObject_GetAttrString( iomodule, "open" );
if( openfunction == NULL ) {
std::cerr << "ERROR: FastFile failed get the io module open function '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
openfile = PyObject_CallFunction( openfunction, "s", filepath,
"s", "r", "i", -1, "s", "UTF8", "s", "replace" );
PyObject* iterfunction = PyObject_GetAttrString( openfile, "__iter__" );
Py_DECREF( openfunction );
if( iterfunction == NULL ) {
std::cerr << "ERROR: FastFile failed get the io module iterator function '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
PyObject* openfileresult = PyObject_CallObject( iterfunction, NULL );
Py_DECREF( iterfunction );
if( openfileresult == NULL ) {
std::cerr << "ERROR: FastFile failed get the io module iterator object '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
fileiterator = PyObject_GetAttrString( openfile, "__next__" );
Py_DECREF( openfileresult );
if( fileiterator == NULL ) {
std::cerr << "ERROR: FastFile failed get the io module iterator object '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
}
~FastFile() {
this->close();
Py_XDECREF( iomodule );
Py_XDECREF( openfile );
Py_XDECREF( fileiterator );
for( PyObject* pyobject : linecache ) {
Py_DECREF( pyobject );
}
}
void close() {
PyObject* closefunction = PyObject_GetAttrString( openfile, "close" );
if( closefunction == NULL ) {
std::cerr << "ERROR: FastFile failed get the close file function for '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
PyObject* closefileresult = PyObject_CallObject( closefunction, NULL );
Py_DECREF( closefunction );
if( closefileresult == NULL ) {
std::cerr << "ERROR: FastFile failed close open file '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
Py_DECREF( closefileresult );
}
bool _getline() {
// Fix StopIteration being raised multiple times because
// _getlines is called multiple times
if( hasfinished ) { return false; }
PyObject* readline = PyObject_CallObject( fileiterator, NULL );
if( readline != NULL ) {
linecount += 1;
linecache.push_back( readline );
return true;
}
// PyErr_Print();
PyErr_Clear();
hasfinished = true;
return false;
}
使用 Visual Studio Compiler 编译时，使用 具有以下性能：
print( 'fastfile_time %.2f%%, python_time %.2f%%' % (
fastfile_time/python_time, python_time/fastfile_time ), flush=True )
$ python3 fastfileperformance.py
Python timedifference 0:00:00.985254
FastFile timedifference 0:00:01.084283
fastfile_time 1.10%, python_time 0.91% = 0.09%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.979861
FastFile timedifference 0:00:01.073879
fastfile_time 1.10%, python_time 0.91% = 0.09%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.990369
FastFile timedifference 0:00:01.086416
fastfile_time 1.10%, python_time 0.91% = 0.09%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.975223
FastFile timedifference 0:00:01.077857
fastfile_time 1.11%, python_time 0.90% = 0.10%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.988327
FastFile timedifference 0:00:01.085866
fastfile_time 1.10%, python_time 0.91% = 0.09%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.971848
FastFile timedifference 0:00:01.087894
fastfile_time 1.12%, python_time 0.89% = 0.11%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.968116
FastFile timedifference 0:00:01.079976
fastfile_time 1.12%, python_time 0.90% = 0.10%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.980856
FastFile timedifference 0:00:01.068325
fastfile_time 1.09%, python_time 0.92% = 0.08%
但是当用g++编译时，它得到了这个性能：
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.703964
FastFile timedifference 0:00:00.813478
fastfile_time 1.16%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.703432
FastFile timedifference 0:00:00.809531
fastfile_time 1.15%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.705319
FastFile timedifference 0:00:00.814130
fastfile_time 1.15%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.711852
FastFile timedifference 0:00:00.837132
fastfile_time 1.18%, python_time 0.85% = 0.15%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.695033
FastFile timedifference 0:00:00.800901
fastfile_time 1.15%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.694661
FastFile timedifference 0:00:00.796754
fastfile_time 1.15%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.699377
FastFile timedifference 0:00:00.816715
fastfile_time 1.17%, python_time 0.86% = 0.14%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.699229
FastFile timedifference 0:00:00.818774
fastfile_time 1.17%, python_time 0.85% = 0.15%
问题是现在，我必须使用 Posix C getline 函数从文件中获取行，然后才将其转换为 Python Unicode 对象使用PyUnicode_DecodeUTF8 并使用我的 caching policy algorithm. This process is for line in file C 实现相比。
如果我从我的代码中删除 PyUnicode_DecodeUTF8 调用，那么，我使用 Posix C getline 的实现变得 5% 比 Python 更快内置 for line in file C 实现。所以，如果我可以让 Python 直接给我一个 Python Unicode String 对象，而不是必须先调用 Posix C getline 函数（然后才转换它的结果到 Python Unicode 对象），我的代码性能将提高几乎 20%（从最大值 23%），即它不会变成 100% 等同于 for line in file 性能，因为我通过缓存东西做了一些工作，但是这种开销很小。
例如，我想采用 _textiowrapper_readline() 函数并在我的代码中使用它，如下所示：
#include <Python.h>
#include <textio.c.h> // C Python file defininig:
// _textiowrapper_readline(),
// CHECK_ATTACHED(),
// PyUnicode_READY(), etc
typedef struct
{
PyObject_HEAD
}
PyMymoduleExtendingPython;
static PyObject*
PyMymoduleExtendingPython_iternext(PyMymoduleExtendingPython* self, PyObject* args)
{
PyObject *line;
CHECK_ATTACHED(self);
line = _textiowrapper_readline(self, -1); // <- function from `textio.c`
if (line == NULL || PyUnicode_READY(line) == -1)
return NULL;
if (PyUnicode_GET_LENGTH(line) == 0) {
/* Reached EOF or would have blocked */
Py_DECREF(line);
Py_CLEAR(self->snapshot);
self->telling = self->seekable;
return NULL;
}
return line;
}
// create my module
PyMODINIT_FUNC PyInit_mymodule_extending_python_api(void)
{
PyObject* mymodule;
PyMymoduleExtendingPython.tp_iternext =
(iternextfunc) PyMymoduleExtendingPython_iternext;
Py_INCREF( &PyMymoduleExtendingPython );
PyModule_AddObject( mymodule, "FastFile", (PyObject*) &PyMymoduleExtendingPython );
return mymodule;
}
我如何包含 C Python 中的 textio 实现并在我自己的 Python C Extension/API 中重用它的代码？
如我在上一个问题
在 PyUnicode_DecodeUTF8 来解码它们，而不是调用 [=16] 来重新实现 Python 算法=] 在我阅读的每一行。
但是，与其重写所有已经 written/done/ready 的 C Python 代码来读取行，我可以调用它的 "getline" 函数 _textiowrapper_readline() 直接获取该行一个 Python Unicode 对象，然后缓存 it/use 就像我已经在处理从 Posix C getline 函数中得到的行一样（并传递给 PyUnicode_DecodeUTF8() 解码它们变成 Python Unicode 对象）。
我没有设法直接导入 C API（扩展）函数，但我使用 Python 导入了 io 模块，它有一个 link/reference到全局内置函数 open 作为 io.open().
bool hasfinished;
const char* filepath;
long long int linecount;
std::deque<PyObject*> linecache;
PyObject* iomodule;
PyObject* openfile;
PyObject* fileiterator;
FastFile(const char* filepath) : hasfinished(false), filepath(filepath), linecount(0) {
iomodule = PyImport_ImportModule( "io" );
if( iomodule == NULL ) {
std::cerr << "ERROR: FastFile failed to import the io module '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
PyObject* openfunction = PyObject_GetAttrString( iomodule, "open" );
if( openfunction == NULL ) {
std::cerr << "ERROR: FastFile failed get the io module open function '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
openfile = PyObject_CallFunction( openfunction, "s", filepath,
"s", "r", "i", -1, "s", "UTF8", "s", "replace" );
PyObject* iterfunction = PyObject_GetAttrString( openfile, "__iter__" );
Py_DECREF( openfunction );
if( iterfunction == NULL ) {
std::cerr << "ERROR: FastFile failed get the io module iterator function '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
PyObject* openfileresult = PyObject_CallObject( iterfunction, NULL );
Py_DECREF( iterfunction );
if( openfileresult == NULL ) {
std::cerr << "ERROR: FastFile failed get the io module iterator object '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
fileiterator = PyObject_GetAttrString( openfile, "__next__" );
Py_DECREF( openfileresult );
if( fileiterator == NULL ) {
std::cerr << "ERROR: FastFile failed get the io module iterator object '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
}
~FastFile() {
this->close();
Py_XDECREF( iomodule );
Py_XDECREF( openfile );
Py_XDECREF( fileiterator );
for( PyObject* pyobject : linecache ) {
Py_DECREF( pyobject );
}
}
void close() {
PyObject* closefunction = PyObject_GetAttrString( openfile, "close" );
if( closefunction == NULL ) {
std::cerr << "ERROR: FastFile failed get the close file function for '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
PyObject* closefileresult = PyObject_CallObject( closefunction, NULL );
Py_DECREF( closefunction );
if( closefileresult == NULL ) {
std::cerr << "ERROR: FastFile failed close open file '"
<< filepath << "')!" << std::endl;
PyErr_Print();
return;
}
Py_DECREF( closefileresult );
}
bool _getline() {
// Fix StopIteration being raised multiple times because
// _getlines is called multiple times
if( hasfinished ) { return false; }
PyObject* readline = PyObject_CallObject( fileiterator, NULL );
if( readline != NULL ) {
linecount += 1;
linecache.push_back( readline );
return true;
}
// PyErr_Print();
PyErr_Clear();
hasfinished = true;
return false;
}
使用 Visual Studio Compiler 编译时，使用
print( 'fastfile_time %.2f%%, python_time %.2f%%' % (
fastfile_time/python_time, python_time/fastfile_time ), flush=True )
$ python3 fastfileperformance.py
Python timedifference 0:00:00.985254
FastFile timedifference 0:00:01.084283
fastfile_time 1.10%, python_time 0.91% = 0.09%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.979861
FastFile timedifference 0:00:01.073879
fastfile_time 1.10%, python_time 0.91% = 0.09%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.990369
FastFile timedifference 0:00:01.086416
fastfile_time 1.10%, python_time 0.91% = 0.09%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.975223
FastFile timedifference 0:00:01.077857
fastfile_time 1.11%, python_time 0.90% = 0.10%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.988327
FastFile timedifference 0:00:01.085866
fastfile_time 1.10%, python_time 0.91% = 0.09%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.971848
FastFile timedifference 0:00:01.087894
fastfile_time 1.12%, python_time 0.89% = 0.11%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.968116
FastFile timedifference 0:00:01.079976
fastfile_time 1.12%, python_time 0.90% = 0.10%
$ python3 fastfileperformance.py
Python timedifference 0:00:00.980856
FastFile timedifference 0:00:01.068325
fastfile_time 1.09%, python_time 0.92% = 0.08%
但是当用g++编译时，它得到了这个性能：
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.703964
FastFile timedifference 0:00:00.813478
fastfile_time 1.16%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.703432
FastFile timedifference 0:00:00.809531
fastfile_time 1.15%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.705319
FastFile timedifference 0:00:00.814130
fastfile_time 1.15%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.711852
FastFile timedifference 0:00:00.837132
fastfile_time 1.18%, python_time 0.85% = 0.15%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.695033
FastFile timedifference 0:00:00.800901
fastfile_time 1.15%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.694661
FastFile timedifference 0:00:00.796754
fastfile_time 1.15%, python_time 0.87% = 0.13%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.699377
FastFile timedifference 0:00:00.816715
fastfile_time 1.17%, python_time 0.86% = 0.14%
$ /bin/python3.6 fastfileperformance.py
Python timedifference 0:00:00.699229
FastFile timedifference 0:00:00.818774
fastfile_time 1.17%, python_time 0.85% = 0.15%