如何在 Cloud Composer 中为 运行 KubernetesPodOperator 设置适当的权限？
How to set proper permissions to run KubernetesPodOperator in Cloud Composer?
我正尝试根据此处的文档在我的 Composer 环境中 运行 一个简单的 KubernetesPodOperator。
气流 运行time 由于用户 "default" 权限不足而失败。
也就是说，如何正确创建环境或设置默认用户权限才能使此代码正常工作？
DAG:
price_analysis = KubernetesPodOperator(
task_id='price-analysis',
name='price-analysis',
namespace='default',
image='bash',
image_pull_policy='Always',
cmds=['echo'],
arguments=['something'],
env_vars={
'EXPOSURE_THRESHOLD': '5',
'ESTIMATE_WINDOW': '3,7',
},
in_cluster=True,
)
日志：
-------------------------------------------------------------------------------
Starting attempt 1 of
-------------------------------------------------------------------------------
[2019-04-03 14:54:15,611] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): price-analysis> on 2019-04-03T14:53:59.658367+00:00
[2019-04-03 14:54:15,612] {base_task_runner.py:118} INFO - Running: ['bash', '-c', u'airflow run vat-analysis price-analysis 2019-04-03T14:53:59.658367+00:00 --job_id 54 --raw -sd DAGS_FOLDER/vat_analysis_dag.py --cfg_path /tmp/tmp3RdZOV']
[2019-04-03 14:54:18,375] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:18,374] {settings.py:176} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2019-04-03 14:54:19,652] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,651] {default_celery.py:80} WARNING - You have configured a result_backend of redis://airflow-redis-service.default.svc.cluster.local:6379/0, it is highly recommended to use an alternative result_backend (i.e. a database).
[2019-04-03 14:54:19,659] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,659] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-04-03 14:54:19,826] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,825] {app.py:51} WARNING - Using default Composer Environment Variables. Overrides have not been applied.
[2019-04-03 14:54:19,842] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,842] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:19,868] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,867] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:20,380] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:20,378] {models.py:271} INFO - Filling up the DagBag from /home/airflow/gcs/dags/vat_analysis_dag.py
[2019-04-03 14:54:21,490] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,490] {cli.py:484} INFO - Running <TaskInstance: vat-analysis.price-analysis 2019-04-03T14:53:59.658367+00:00 [running]> on host airflow-worker-5b6d7c75c9-w6995
[2019-04-03 14:54:22,093] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,822] {pod_launcher.py:58} ERROR - Exception when attempting to create Namespaced Pod.
[2019-04-03 14:54:22,103] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis Traceback (most recent call last):
[2019-04-03 14:54:22,107] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 55, in run_pod_async
[2019-04-03 14:54:22,113] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
[2019-04-03 14:54:22,116] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6115, in create_namespaced_pod
[2019-04-03 14:54:22,122] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
[2019-04-03 14:54:22,126] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6206, in create_namespaced_pod_with_http_info
[2019-04-03 14:54:22,129] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis collection_formats=collection_formats)
[2019-04-03 14:54:22,134] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 321, in call_api
[2019-04-03 14:54:22,150] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis _return_http_data_only, collection_formats, _preload_content, _request_timeout)
[2019-04-03 14:54:22,155] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 155, in __call_api
[2019-04-03 14:54:22,159] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis _request_timeout=_request_timeout)
[2019-04-03 14:54:22,138] {models.py:1760} ERROR - (403
Reason: Forbidde
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 03 Apr 2019 14:54:21 GMT', 'Audit-Id': 'c027d4cb-5186-498a-a9b5-0e6c4420b816', 'Content-Length': '284', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:composer-1-6-0-airflow-1-10-1-ea0745b4:default\" cannot create pods in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403
我从 Google 组中的 Google Composer 讨论组收到了回复。
一个细节：服务帐户 default:default 必须与您在错误消息中看到的服务帐户相同（在我的例子中是 composer-1-6-0-airflow-1-10-1-ea0745b4:default）。
CLUSTER_NAME=.....
NAMESPACE=k8s-tasks
kubectl create ns ${NAMESPACE}
kubectl create clusterrolebinding default-admin \
--clusterrole cluster-admin \
--serviceaccount=default:default \
--namespace ${NAMESPACE}
我正尝试根据此处的文档在我的 Composer 环境中 运行 一个简单的 KubernetesPodOperator。
气流 运行time 由于用户 "default" 权限不足而失败。
也就是说，如何正确创建环境或设置默认用户权限才能使此代码正常工作？
DAG:
price_analysis = KubernetesPodOperator(
task_id='price-analysis',
name='price-analysis',
namespace='default',
image='bash',
image_pull_policy='Always',
cmds=['echo'],
arguments=['something'],
env_vars={
'EXPOSURE_THRESHOLD': '5',
'ESTIMATE_WINDOW': '3,7',
},
in_cluster=True,
)
日志：
-------------------------------------------------------------------------------
Starting attempt 1 of
-------------------------------------------------------------------------------
[2019-04-03 14:54:15,611] {models.py:1595} INFO - Executing <Task(KubernetesPodOperator): price-analysis> on 2019-04-03T14:53:59.658367+00:00
[2019-04-03 14:54:15,612] {base_task_runner.py:118} INFO - Running: ['bash', '-c', u'airflow run vat-analysis price-analysis 2019-04-03T14:53:59.658367+00:00 --job_id 54 --raw -sd DAGS_FOLDER/vat_analysis_dag.py --cfg_path /tmp/tmp3RdZOV']
[2019-04-03 14:54:18,375] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:18,374] {settings.py:176} INFO - setting.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800
[2019-04-03 14:54:19,652] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,651] {default_celery.py:80} WARNING - You have configured a result_backend of redis://airflow-redis-service.default.svc.cluster.local:6379/0, it is highly recommended to use an alternative result_backend (i.e. a database).
[2019-04-03 14:54:19,659] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,659] {__init__.py:51} INFO - Using executor CeleryExecutor
[2019-04-03 14:54:19,826] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,825] {app.py:51} WARNING - Using default Composer Environment Variables. Overrides have not been applied.
[2019-04-03 14:54:19,842] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,842] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:19,868] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:19,867] {configuration.py:516} INFO - Reading the config from /etc/airflow/airflow.cfg
[2019-04-03 14:54:20,380] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:20,378] {models.py:271} INFO - Filling up the DagBag from /home/airflow/gcs/dags/vat_analysis_dag.py
[2019-04-03 14:54:21,490] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,490] {cli.py:484} INFO - Running <TaskInstance: vat-analysis.price-analysis 2019-04-03T14:53:59.658367+00:00 [running]> on host airflow-worker-5b6d7c75c9-w6995
[2019-04-03 14:54:22,093] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis [2019-04-03 14:54:21,822] {pod_launcher.py:58} ERROR - Exception when attempting to create Namespaced Pod.
[2019-04-03 14:54:22,103] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis Traceback (most recent call last):
[2019-04-03 14:54:22,107] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/airflow/airflow/contrib/kubernetes/pod_launcher.py", line 55, in run_pod_async
[2019-04-03 14:54:22,113] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis resp = self._client.create_namespaced_pod(body=req, namespace=pod.namespace)
[2019-04-03 14:54:22,116] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6115, in create_namespaced_pod
[2019-04-03 14:54:22,122] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis (data) = self.create_namespaced_pod_with_http_info(namespace, body, **kwargs)
[2019-04-03 14:54:22,126] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/apis/core_v1_api.py", line 6206, in create_namespaced_pod_with_http_info
[2019-04-03 14:54:22,129] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis collection_formats=collection_formats)
[2019-04-03 14:54:22,134] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 321, in call_api
[2019-04-03 14:54:22,150] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis _return_http_data_only, collection_formats, _preload_content, _request_timeout)
[2019-04-03 14:54:22,155] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis File "/usr/local/lib/python2.7/dist-packages/kubernetes/client/api_client.py", line 155, in __call_api
[2019-04-03 14:54:22,159] {base_task_runner.py:101} INFO - Job 54: Subtask price-analysis _request_timeout=_request_timeout)
[2019-04-03 14:54:22,138] {models.py:1760} ERROR - (403
Reason: Forbidde
HTTP response headers: HTTPHeaderDict({'Date': 'Wed, 03 Apr 2019 14:54:21 GMT', 'Audit-Id': 'c027d4cb-5186-498a-a9b5-0e6c4420b816', 'Content-Length': '284', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff'}
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods is forbidden: User \"system:serviceaccount:composer-1-6-0-airflow-1-10-1-ea0745b4:default\" cannot create pods in the namespace \"default\"","reason":"Forbidden","details":{"kind":"pods"},"code":403
我从 Google 组中的 Google Composer 讨论组收到了回复。
一个细节：服务帐户 default:default 必须与您在错误消息中看到的服务帐户相同（在我的例子中是 composer-1-6-0-airflow-1-10-1-ea0745b4:default）。
CLUSTER_NAME=.....
NAMESPACE=k8s-tasks
kubectl create ns ${NAMESPACE}
kubectl create clusterrolebinding default-admin \
--clusterrole cluster-admin \
--serviceaccount=default:default \
--namespace ${NAMESPACE}