docker-撰写未知MySQL服务器主机'mysql'（气流）
docker-compose Unknown MySQL server host 'mysql' (airflow)
我尝试创建我的 Airflow 服务。当我 运行 三个 docker 分别（没有 docker-compose）时，一切正常，但是当我尝试使用 docker-compose 做同样的事情时，我得到了错误。似乎气流没有看到数据库。
docker-compose.yml
version: '2'
services:
mysql:
image: "someregisty/mysql"
environment:
- MYSQL_ROOT_PASSWORD=somepassword
redis:
image: "someregisty/redis"
airflow:
image: "someregisty/airflow"
volumes:
- ./dags:/home/airflow/dags
- ./logs:/home/airflow/logs
- ./config/airflow.cfg:/home/airflow/airflow.cfg
- ./config/resolv.conf:/etc/resolv.conf
- ./config/somekeytab.keytab:/home/airflow/airflow.keytab
ports:
- "8484:8080"
environment:
- AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
- AIRFLOW__CELERY__CELERY_RESULT_BACKEND=db+mysql://root:somepassword@mysql/airflow
- AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://root:somepassword@mysql/airflow
- AIRFLOW__CORE__AIRFLOW_HOME=/home/airflow
- AIRFLOW__CORE__DAGS_FOLDER=/home/airflow/dags/
- AIRFLOW__CORE__PLUGINS_FOLDER=/home/airflow/plugins/
- AIRFLOW_SCHEDULER=true
- AIRFLOW_WEBSERVER=true
- AIRFLOW_WORKER=true
- AIRFLOW_KERBEROS=true
- AIRFLOW_KERBEROS_PRINCIPAL=prj_bdc@DCWP.PL
- AIRFLOW__KERBEROS__CCACHE=/home/airflow/kerberos.keytab.cache
- KRB5CCNAME=/home/airflow/kerberos.keytab.cache
depends_on:
- redis
- mysql
我运行
docker-compose up -d
docker-compose exec mysql mysql -u root --password=somepassword -e "create database airflow"
一切正常。那我运行
docker-compose exec airflow airflow initdb
然后我得到
[2017-09-28 16:34:36,100] {__init__.py:57} INFO - Using executor CeleryExecutor
/usr/lib/python2.7/site-packages/airflow/utils/helpers.py:406: DeprecationWarning: Importing BaseSensorOperator directly from <module 'airflow.operators' from '/usr/lib/python2.7/site-packages/airflow/operators/__init__.pyc'> has been deprecated. Please import from '<module 'airflow.operators' from '/usr/lib/python2.7/site-packages/airflow/operators/__init__.pyc'>.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
DB: mysql://root:***@mysql/airflow
[2017-09-28 16:34:36,666] {db.py:287} INFO - Creating tables
Traceback (most recent call last):
File "/usr/bin/airflow", line 28, in <module>
args.func(args)
File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 951, in initdb
db_utils.initdb()
File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 106, in initdb
upgradedb()
File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 294, in upgradedb
command.upgrade(config, 'heads')
File "/usr/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade
script.run_env()
File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 416, in run_env
util.load_python_file(self.dir, 'env.py')
File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "/usr/lib/python2.7/site-packages/airflow/migrations/env.py", line 86, in <module>
run_migrations_online()
File "/usr/lib/python2.7/site-packages/airflow/migrations/env.py", line 73, in run_migrations_online
with connectable.connect() as connection:
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2091, in connect
return self._connection_cls(self, **kwargs)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 90, in __init__
if connection is not None else engine.raw_connection()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2177, in raw_connection
self.pool.unique_connection, _connection)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
e, dialect, self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1465, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
return fn()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 328, in unique_connection
return _ConnectionFairy._checkout(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
rec = pool._do_get()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
self._dec_overflow()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
return self._create_connection()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
return _ConnectionRecord(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
self.__connect(first_connect_check=True)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 651, in __connect
connection = pool._invoke_creator(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 86, in Connect
return Connection(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 204, in __init__
super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2005, "Unknown MySQL server host 'mysql' (0)")
不知道怎么回事。也许你能帮我？ :)
docker 和 docker-compose 的版本是
Docker version 1.12.6, build 96d83a5/1.12.6
docker-compose version 1.9.0, build 2585387
所以这个错误有点误导。连接失败是因为数据库不存在，而不是 mysql 主机不存在。
将您的 mysql 服务更改为以下
mysql:
image: "someregisty/mysql"
environment:
- MYSQL_ROOT_PASSWORD=somepassword
- MYSQL_DATABASE=airflow
问题是由将 resolv.conf 文件绑定到气流容器引起的。您从气流容器中收到 unknown MySQL server host 'mysql' 错误的原因是 mysql 主机名未解析为气流容器的 IP。
解决方案 1
从 Compose 配置中删除 ./config/resolv.conf:/etc/resolv.conf。它将导致 Docker 正确解析主机名。如果您需要指定额外的 DNS 服务器，请使用 Compose --dns 选项 documented here.
方案二（不推荐）
将 nameserver 127.0.0.11 行添加到 /etc/resolv.conf 文件。它允许使用 Docker 嵌入式 DNS 服务器，这是默认设置。您也可以将自定义条目保存在该文件中。请注意，此解决方案是一种 hack，因为如文档中所述，Docker 负责管理 /etc/resolv.conf 文件。引用文档来支持它：
So you should not assume the way the files such as /etc/hosts,
/etc/resolv.conf are managed inside the containers and leave the files
alone and use the following Docker options instead
更多
您可以阅读有关 Docker embedded DNS server in user-defined networks here 的更多信息。
我尝试创建我的 Airflow 服务。当我 运行 三个 docker 分别（没有 docker-compose）时，一切正常，但是当我尝试使用 docker-compose 做同样的事情时，我得到了错误。似乎气流没有看到数据库。
docker-compose.yml
version: '2'
services:
mysql:
image: "someregisty/mysql"
environment:
- MYSQL_ROOT_PASSWORD=somepassword
redis:
image: "someregisty/redis"
airflow:
image: "someregisty/airflow"
volumes:
- ./dags:/home/airflow/dags
- ./logs:/home/airflow/logs
- ./config/airflow.cfg:/home/airflow/airflow.cfg
- ./config/resolv.conf:/etc/resolv.conf
- ./config/somekeytab.keytab:/home/airflow/airflow.keytab
ports:
- "8484:8080"
environment:
- AIRFLOW__CELERY__BROKER_URL=redis://redis:6379/0
- AIRFLOW__CELERY__CELERY_RESULT_BACKEND=db+mysql://root:somepassword@mysql/airflow
- AIRFLOW__CORE__SQL_ALCHEMY_CONN=mysql://root:somepassword@mysql/airflow
- AIRFLOW__CORE__AIRFLOW_HOME=/home/airflow
- AIRFLOW__CORE__DAGS_FOLDER=/home/airflow/dags/
- AIRFLOW__CORE__PLUGINS_FOLDER=/home/airflow/plugins/
- AIRFLOW_SCHEDULER=true
- AIRFLOW_WEBSERVER=true
- AIRFLOW_WORKER=true
- AIRFLOW_KERBEROS=true
- AIRFLOW_KERBEROS_PRINCIPAL=prj_bdc@DCWP.PL
- AIRFLOW__KERBEROS__CCACHE=/home/airflow/kerberos.keytab.cache
- KRB5CCNAME=/home/airflow/kerberos.keytab.cache
depends_on:
- redis
- mysql
我运行
docker-compose up -d
docker-compose exec mysql mysql -u root --password=somepassword -e "create database airflow"
一切正常。那我运行
docker-compose exec airflow airflow initdb
然后我得到
[2017-09-28 16:34:36,100] {__init__.py:57} INFO - Using executor CeleryExecutor
/usr/lib/python2.7/site-packages/airflow/utils/helpers.py:406: DeprecationWarning: Importing BaseSensorOperator directly from <module 'airflow.operators' from '/usr/lib/python2.7/site-packages/airflow/operators/__init__.pyc'> has been deprecated. Please import from '<module 'airflow.operators' from '/usr/lib/python2.7/site-packages/airflow/operators/__init__.pyc'>.[operator_module]' instead. Support for direct imports will be dropped entirely in Airflow 2.0.
DeprecationWarning)
DB: mysql://root:***@mysql/airflow
[2017-09-28 16:34:36,666] {db.py:287} INFO - Creating tables
Traceback (most recent call last):
File "/usr/bin/airflow", line 28, in <module>
args.func(args)
File "/usr/lib/python2.7/site-packages/airflow/bin/cli.py", line 951, in initdb
db_utils.initdb()
File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 106, in initdb
upgradedb()
File "/usr/lib/python2.7/site-packages/airflow/utils/db.py", line 294, in upgradedb
command.upgrade(config, 'heads')
File "/usr/lib/python2.7/site-packages/alembic/command.py", line 174, in upgrade
script.run_env()
File "/usr/lib/python2.7/site-packages/alembic/script/base.py", line 416, in run_env
util.load_python_file(self.dir, 'env.py')
File "/usr/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 93, in load_python_file
module = load_module_py(module_id, path)
File "/usr/lib/python2.7/site-packages/alembic/util/compat.py", line 79, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "/usr/lib/python2.7/site-packages/airflow/migrations/env.py", line 86, in <module>
run_migrations_online()
File "/usr/lib/python2.7/site-packages/airflow/migrations/env.py", line 73, in run_migrations_online
with connectable.connect() as connection:
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2091, in connect
return self._connection_cls(self, **kwargs)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 90, in __init__
if connection is not None else engine.raw_connection()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2177, in raw_connection
self.pool.unique_connection, _connection)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2151, in _wrap_pool_connect
e, dialect, self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1465, in _handle_dbapi_exception_noconnection
exc_info
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2147, in _wrap_pool_connect
return fn()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 328, in unique_connection
return _ConnectionFairy._checkout(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 766, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 516, in checkout
rec = pool._do_get()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1138, in _do_get
self._dec_overflow()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1135, in _do_get
return self._create_connection()
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 333, in _create_connection
return _ConnectionRecord(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 461, in __init__
self.__connect(first_connect_check=True)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 651, in __connect
connection = pool._invoke_creator(self)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
return dialect.connect(*cargs, **cparams)
File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 393, in connect
return self.dbapi.connect(*cargs, **cparams)
File "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 86, in Connect
return Connection(*args, **kwargs)
File "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 204, in __init__
super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (2005, "Unknown MySQL server host 'mysql' (0)")
不知道怎么回事。也许你能帮我？ :)
docker 和 docker-compose 的版本是
Docker version 1.12.6, build 96d83a5/1.12.6
docker-compose version 1.9.0, build 2585387
所以这个错误有点误导。连接失败是因为数据库不存在，而不是 mysql 主机不存在。
将您的 mysql 服务更改为以下
mysql:
image: "someregisty/mysql"
environment:
- MYSQL_ROOT_PASSWORD=somepassword
- MYSQL_DATABASE=airflow
问题是由将 resolv.conf 文件绑定到气流容器引起的。您从气流容器中收到 unknown MySQL server host 'mysql' 错误的原因是 mysql 主机名未解析为气流容器的 IP。
解决方案 1
从 Compose 配置中删除 ./config/resolv.conf:/etc/resolv.conf。它将导致 Docker 正确解析主机名。如果您需要指定额外的 DNS 服务器，请使用 Compose --dns 选项 documented here.
方案二（不推荐）
将 nameserver 127.0.0.11 行添加到 /etc/resolv.conf 文件。它允许使用 Docker 嵌入式 DNS 服务器，这是默认设置。您也可以将自定义条目保存在该文件中。请注意，此解决方案是一种 hack，因为如文档中所述，Docker 负责管理 /etc/resolv.conf 文件。引用文档来支持它：
So you should not assume the way the files such as /etc/hosts, /etc/resolv.conf are managed inside the containers and leave the files alone and use the following Docker options instead
更多
您可以阅读有关 Docker embedded DNS server in user-defined networks here 的更多信息。