将 Cythonization 添加到 setup.py 后，Travis 日志颜色消失
Travis log colors disappears when Cythonization is added to setup.py
从 https://travis-ci.org/nltk/nltk/builds/529221349，我们看到几个构建和所有其他构建的日志在 运行 tox 时带有 passed/failed/skip 颜色，例如
对 py-travis 环境使用 tox.ini 配置：
[tox]
envlist =
py{27,35,36,37}
pypy
py{27,35,36}-nodeps
py{27,35,36}-jenkins
py-cythonized
py-travis
[testenv]
; simplify numpy installation
setenv =
LAPACK=
ATLAS=None
PYTHONWARNINGS=ignore
; Copy all environment variables to the tox test environment
passenv = *
deps =
numpy
nose >= 1.2.1
coverage
text-unidecode
twython
pyparsing
python-crfsuite
rednose
changedir = nltk/test
commands =
; scipy and scikit-learn requires numpy even to run setup.py so
; they can't be installed in one command
pip install scipy scikit-learn
; python runtests.py --with-coverage --cover-inclusive --cover-package=nltk --cover-html --cover-html-dir={envdir}/docs []
python runtests.py []
commands =
python runtests.py []
[testenv:py-travis]
extras = all
setenv =
NLTK_DATA = {homedir}/nltk_data/
commands = {toxinidir}/tools/travis/coverage-pylint.sh
但是当 setup.py 的 cythonization 启动时，tox 和 travis 配置似乎相同，
[testenv:py-cythonized]
deps =
Cython >= 0.28.5
setenv =
CYTHONIZE_NLTK = true
NLTK_DATA = {homedir}/nltk_data/
extras = all
commands = {toxinidir}/tools/travis/coverage-pylint.sh
运行构建时，颜色消失：
安装 setup.py 对于 py-travis 和 py-cynthonized 构建完全相同：
# Work around mbcs bug in distutils.
# http://bugs.python.org/issue10945
import codecs
try:
codecs.lookup('mbcs')
except LookupError:
ascii = codecs.lookup('ascii')
func = lambda name, enc=ascii: {True: enc}.get(name == 'mbcs')
codecs.register(func)
import os
# Use the VERSION file to get NLTK version
version_file = os.path.join(os.path.dirname(__file__), 'nltk', 'VERSION')
with open(version_file) as fh:
nltk_version = fh.read().strip()
# setuptools
from setuptools import setup, find_packages
# Specify groups of optional dependencies
extras_require = {
'machine_learning': ['gensim', 'numpy', 'python-crfsuite', 'scikit-learn', 'scipy'],
'plot': ['matplotlib'],
'tgrep': ['pyparsing'],
'twitter': ['twython'],
'corenlp': ['requests'],
}
# Add a group made up of all optional dependencies
extras_require['all'] = set(
package for group in extras_require.values() for package in group
)
MODULES_TO_COMPILE = [
'nltk.grammar',
'nltk.parse.chart',
'nltk.tokenize.*',
'nltk.probability',
'nltk.util',
'nltk.stem.*',
'nltk.lm.*',
'nltk.translate.*',
'nltk.tbl.*',
'nltk.sentiment.*',
'nltk.cluster.*',
'nltk.classify.*',
'nltk.metrics.*',
'nltk.chunk.*',
'nltk.sem.*',
]
def compile_modules(modules):
"""
Compile the named modules using Cython, using the clearer Python 3 semantics.
"""
import Cython
from Cython.Build import cythonize
files = [name.replace('.', os.path.sep) + '.py' for name in modules]
print("Compiling %d modules using Cython %s" % (len(modules), Cython.__version__))
return cythonize(files, language_level=3)
if os.getenv('CYTHONIZE_NLTK') == 'true':
ext_modules = compile_modules(MODULES_TO_COMPILE)
else:
ext_modules = None
setup(
name="nltk",
description="Natural Language Toolkit",
version=nltk_version,
url="http://nltk.org/",
long_description="""\
The Natural Language Toolkit (NLTK) is a Python package for
natural language processing. NLTK requires Python 2.7, 3.5, 3.6, or 3.7.""",
license="Apache License, Version 2.0",
keywords=[
'NLP',
'CL',
'natural language processing',
'computational linguistics',
'parsing',
'tagging',
'tokenizing',
'syntax',
'linguistics',
'language',
'natural language',
'text analytics',
],
maintainer="Steven Bird",
maintainer_email="stevenbird1@gmail.com",
author="Steven Bird",
author_email="stevenbird1@gmail.com",
classifiers=[
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Intended Audience :: Education',
'Intended Audience :: Information Technology',
'Intended Audience :: Science/Research',
'License :: OSI Approved :: Apache Software License',
'Operating System :: OS Independent',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Topic :: Scientific/Engineering',
'Topic :: Scientific/Engineering :: Artificial Intelligence',
'Topic :: Scientific/Engineering :: Human Machine Interfaces',
'Topic :: Scientific/Engineering :: Information Analysis',
'Topic :: Text Processing',
'Topic :: Text Processing :: Filters',
'Topic :: Text Processing :: General',
'Topic :: Text Processing :: Indexing',
'Topic :: Text Processing :: Linguistic',
],
package_data={'nltk': ['test/*.doctest', 'VERSION']},
install_requires=[
'six',
'singledispatch; python_version < "3.4"'
],
extras_require=extras_require,
packages=find_packages(),
ext_modules=ext_modules,
zip_safe=False, # since normal files will be present too?
)
为什么 Cythonized 版本的颜色消失了？
如何为 Cythonized 构建启用颜色？
对于某些背景，代码来自 nltk 库，cynthonization tests/builds 的完整分支在 https://github.com/alvations/nltk/tree/cythonize
Why does the color disappear for the Cythonized builds?
因为 py-cythonized 环境没有安装 nose（这就是为什么测试 运行 首先是 stdlib 的 unittest）和 rednose（鼻子输出着色）。
发生这种情况是因为环境 deps 被覆盖了。 py-travis 没有声明任何自己的依赖项，因此它从全局 testenv 配置中继承了 deps 设置。 py-cythonized 需要 Cython，因此它重新定义了 deps 列表，丢失了测试执行所需的所有包。
How to enable the color for the Cythonized builds?
将依赖项从全局 testenv 复制到 py-cythonized。提议的补丁：
diff --git a/tox.ini b/tox.ini
index a267d9a5a..41740e19b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -133,6 +133,14 @@ commands =
[testenv:py-cythonized]
deps =
Cython >= 0.28.5
+ numpy
+ nose >= 1.2.1
+ coverage
+ text-unidecode
+ twython
+ pyparsing
+ python-crfsuite
+ rednose
setenv =
CYTHONIZE_NLTK = true
NLTK_DATA = {homedir}/nltk_data/
应用补丁后，测试输出的颜色很漂亮：example run on travis
更新：使用条件设置
为避免重复依赖，可以使用条件依赖：
[testenv]
deps =
rednose # this one is global dependency
py-cythonized: cython # this one is specific for testenv:py-cythonized
建议的补丁：
diff --git a/tox.ini b/tox.ini
index a267d9a5a..fa0839b96 100644
--- a/tox.ini
+++ b/tox.ini
@@ -26,6 +26,7 @@ deps =
pyparsing
python-crfsuite
rednose
+ py-cythonized: Cython >= 0.28.5
changedir = nltk/test
commands =
@@ -131,8 +132,6 @@ commands =
# Test Cython compiled installation.
[testenv:py-cythonized]
-deps =
- Cython >= 0.28.5
setenv =
CYTHONIZE_NLTK = true
NLTK_DATA = {homedir}/nltk_data/
从 https://travis-ci.org/nltk/nltk/builds/529221349，我们看到几个构建和所有其他构建的日志在 运行 tox 时带有 passed/failed/skip 颜色，例如
对 py-travis 环境使用 tox.ini 配置：
[tox]
envlist =
py{27,35,36,37}
pypy
py{27,35,36}-nodeps
py{27,35,36}-jenkins
py-cythonized
py-travis
[testenv]
; simplify numpy installation
setenv =
LAPACK=
ATLAS=None
PYTHONWARNINGS=ignore
; Copy all environment variables to the tox test environment
passenv = *
deps =
numpy
nose >= 1.2.1
coverage
text-unidecode
twython
pyparsing
python-crfsuite
rednose
changedir = nltk/test
commands =
; scipy and scikit-learn requires numpy even to run setup.py so
; they can't be installed in one command
pip install scipy scikit-learn
; python runtests.py --with-coverage --cover-inclusive --cover-package=nltk --cover-html --cover-html-dir={envdir}/docs []
python runtests.py []
commands =
python runtests.py []
[testenv:py-travis]
extras = all
setenv =
NLTK_DATA = {homedir}/nltk_data/
commands = {toxinidir}/tools/travis/coverage-pylint.sh
但是当 setup.py 的 cythonization 启动时，tox 和 travis 配置似乎相同，
[testenv:py-cythonized]
deps =
Cython >= 0.28.5
setenv =
CYTHONIZE_NLTK = true
NLTK_DATA = {homedir}/nltk_data/
extras = all
commands = {toxinidir}/tools/travis/coverage-pylint.sh
运行构建时，颜色消失：
安装 setup.py 对于 py-travis 和 py-cynthonized 构建完全相同：
# Work around mbcs bug in distutils.
# http://bugs.python.org/issue10945
import codecs
try:
codecs.lookup('mbcs')
except LookupError:
ascii = codecs.lookup('ascii')
func = lambda name, enc=ascii: {True: enc}.get(name == 'mbcs')
codecs.register(func)
import os
# Use the VERSION file to get NLTK version
version_file = os.path.join(os.path.dirname(__file__), 'nltk', 'VERSION')
with open(version_file) as fh:
nltk_version = fh.read().strip()
# setuptools
from setuptools import setup, find_packages
# Specify groups of optional dependencies
extras_require = {
'machine_learning': ['gensim', 'numpy', 'python-crfsuite', 'scikit-learn', 'scipy'],
'plot': ['matplotlib'],
'tgrep': ['pyparsing'],
'twitter': ['twython'],
'corenlp': ['requests'],
}
# Add a group made up of all optional dependencies
extras_require['all'] = set(
package for group in extras_require.values() for package in group
)
MODULES_TO_COMPILE = [
'nltk.grammar',
'nltk.parse.chart',
'nltk.tokenize.*',
'nltk.probability',
'nltk.util',
'nltk.stem.*',
'nltk.lm.*',
'nltk.translate.*',
'nltk.tbl.*',
'nltk.sentiment.*',
'nltk.cluster.*',
'nltk.classify.*',
'nltk.metrics.*',
'nltk.chunk.*',
'nltk.sem.*',
]
def compile_modules(modules):
"""
Compile the named modules using Cython, using the clearer Python 3 semantics.
"""
import Cython
from Cython.Build import cythonize
files = [name.replace('.', os.path.sep) + '.py' for name in modules]
print("Compiling %d modules using Cython %s" % (len(modules), Cython.__version__))
return cythonize(files, language_level=3)
if os.getenv('CYTHONIZE_NLTK') == 'true':
ext_modules = compile_modules(MODULES_TO_COMPILE)
else:
ext_modules = None
setup(
name="nltk",
description="Natural Language Toolkit",
version=nltk_version,
url="http://nltk.org/",
long_description="""\
The Natural Language Toolkit (NLTK) is a Python package for
natural language processing. NLTK requires Python 2.7, 3.5, 3.6, or 3.7.""",
license="Apache License, Version 2.0",
keywords=[
'NLP',
'CL',
'natural language processing',
'computational linguistics',
'parsing',
'tagging',
'tokenizing',
'syntax',
'linguistics',
'language',
'natural language',
'text analytics',
],
maintainer="Steven Bird",
maintainer_email="stevenbird1@gmail.com",
author="Steven Bird",
author_email="stevenbird1@gmail.com",
classifiers=[
'Development Status :: 5 - Production/Stable',
'Intended Audience :: Developers',
'Intended Audience :: Education',
'Intended Audience :: Information Technology',
'Intended Audience :: Science/Research',
'License :: OSI Approved :: Apache Software License',
'Operating System :: OS Independent',
'Programming Language :: Python :: 2.7',
'Programming Language :: Python :: 3.5',
'Programming Language :: Python :: 3.6',
'Programming Language :: Python :: 3.7',
'Topic :: Scientific/Engineering',
'Topic :: Scientific/Engineering :: Artificial Intelligence',
'Topic :: Scientific/Engineering :: Human Machine Interfaces',
'Topic :: Scientific/Engineering :: Information Analysis',
'Topic :: Text Processing',
'Topic :: Text Processing :: Filters',
'Topic :: Text Processing :: General',
'Topic :: Text Processing :: Indexing',
'Topic :: Text Processing :: Linguistic',
],
package_data={'nltk': ['test/*.doctest', 'VERSION']},
install_requires=[
'six',
'singledispatch; python_version < "3.4"'
],
extras_require=extras_require,
packages=find_packages(),
ext_modules=ext_modules,
zip_safe=False, # since normal files will be present too?
)
为什么 Cythonized 版本的颜色消失了？
如何为 Cythonized 构建启用颜色？
对于某些背景，代码来自 nltk 库，cynthonization tests/builds 的完整分支在 https://github.com/alvations/nltk/tree/cythonize
Why does the color disappear for the Cythonized builds?
因为 py-cythonized 环境没有安装 nose（这就是为什么测试 运行 首先是 stdlib 的 unittest）和 rednose（鼻子输出着色）。
发生这种情况是因为环境 deps 被覆盖了。 py-travis 没有声明任何自己的依赖项，因此它从全局 testenv 配置中继承了 deps 设置。 py-cythonized 需要 Cython，因此它重新定义了 deps 列表，丢失了测试执行所需的所有包。
How to enable the color for the Cythonized builds?
将依赖项从全局 testenv 复制到 py-cythonized。提议的补丁：
diff --git a/tox.ini b/tox.ini
index a267d9a5a..41740e19b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -133,6 +133,14 @@ commands =
[testenv:py-cythonized]
deps =
Cython >= 0.28.5
+ numpy
+ nose >= 1.2.1
+ coverage
+ text-unidecode
+ twython
+ pyparsing
+ python-crfsuite
+ rednose
setenv =
CYTHONIZE_NLTK = true
NLTK_DATA = {homedir}/nltk_data/
应用补丁后，测试输出的颜色很漂亮：example run on travis
更新：使用条件设置
为避免重复依赖，可以使用条件依赖：
[testenv]
deps =
rednose # this one is global dependency
py-cythonized: cython # this one is specific for testenv:py-cythonized
建议的补丁：
diff --git a/tox.ini b/tox.ini
index a267d9a5a..fa0839b96 100644
--- a/tox.ini
+++ b/tox.ini
@@ -26,6 +26,7 @@ deps =
pyparsing
python-crfsuite
rednose
+ py-cythonized: Cython >= 0.28.5
changedir = nltk/test
commands =
@@ -131,8 +132,6 @@ commands =
# Test Cython compiled installation.
[testenv:py-cythonized]
-deps =
- Cython >= 0.28.5
setenv =
CYTHONIZE_NLTK = true
NLTK_DATA = {homedir}/nltk_data/