cloud-sql-GCP Dataproc 代理失败,nc:连接到本地主机端口 3306 (tcp) 失败:连接被拒绝
cloud-sql-proxy for GCP Dataproc fails, nc: connect to localhost port 3306 (tcp) failed: Connection refused
我正在使用 Terraform 创建一个数据处理集群,该集群使用 GCP 云sql 实例作为 hivemetastore,terrafrm 项目创建集群及其所有先决条件(网络、服务帐户、云sql 实例和用户等)。
提供 cloud-sql-proxy.sh
来帮助解决这个问题,但是我无法让它工作,当集群创建时 cloud-sql-proxy.sh
失败并出现错误:
nc: connect to localhost port 3306 (tcp) failed: Connection refused
我用头撞墙试图找出原因,但无法深入了解所以希望这里有人可以提供帮助。
我在 https://github.com/jamiekt/democratising-dataproc 主持了 terraform 项目。重现问题非常简单,按照以下步骤操作:
- Install terraform 如果你还没有
- 安装
gcloud
如果您还没有安装
- 创建一个新的 GCP 项目
- 为您的新项目启用Cloud Dataproc API
gcloud auth application-default login #creates a file containing credentials that terraform will use
git clone git@github.com:jamiekt/democratising-dataproc.git && cd democratising-dataproc
export GCP_PROJECT=name-of-project-you-just-created
make init
make apply
应该成功启动一个网络、子网、云sql实例、几个存储桶(其中一个包含云-sql-proxy.sh)、一个服务帐户,然后防火墙在尝试创建 dataproc 集群时失败。
如果有人能看一看并告诉我为什么失败,我将不胜感激。
看来您使用的不是最新版本的cloud-sql-proxy.sh script in the cloud-sql-proxy.sh.tmpl template (diff).
您可能想尝试使用来自 Dataproc Cloud SQL I/O and Hive Metastore 初始化操作的最新脚本版本更新您的模板。
这里有一些问题现在已经解决了:
- service_account_scopes needed specifying and
hive:hive.metastore.warehouse.dir
property needed setting
- Terraform removes the default root user from MySQL so it has to be recreated. I was missing (amongst other things) the host attribute (
host = '%'
)
- 其他各种事情
state of the repo at the time of posting this message 将按预期工作(即使用 Terraform 创建一个使用共享配置单元元存储的数据处理集群)。
感谢@igor-dvorzhak 的回复,您对配置 Hive Metastore 以使用云的文章 link SQL 让我走上了正确的轨道..
按照 here
中的教程,我遇到了完全相同的问题
我看不出你给出的修复是什么。
似乎配置单元尝试通过本地主机访问数据库我不明白为什么
这是来自 dataproc 的日志:
Synchronizing state of mysql.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable mysql
Removed /etc/systemd/system/multi-user.target.wants/mysql.service.
2020-10-07 14:05:49 URL:https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 [16361206/16361206] -> "cloud_sql_proxy.linux.amd64" [1]
Created symlink /etc/systemd/system/multi-user.target.wants/cloud-sql-proxy.service → /usr/lib/systemd/system/cloud-sql-proxy.service.
About to run 'nc -zv localhost 3306' with retries...
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 1 failed! Sleeping 1.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 2 failed! Sleeping 1.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 3 failed! Sleeping 2.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 4 failed! Sleeping 3.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 5 failed! Sleeping 5.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 6 failed! Sleeping 8.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 7 failed! Sleeping 13.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 8 failed! Sleeping 21.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 9 failed! Sleeping 34.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 10 failed! Sleeping 55.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 11 failed! Sleeping 89.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 12 failed! Sleeping 144.
Final attempt of 'nc -zv localhost 3306'...
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
我正在使用 Terraform 创建一个数据处理集群,该集群使用 GCP 云sql 实例作为 hivemetastore,terrafrm 项目创建集群及其所有先决条件(网络、服务帐户、云sql 实例和用户等)。
提供 cloud-sql-proxy.sh
来帮助解决这个问题,但是我无法让它工作,当集群创建时 cloud-sql-proxy.sh
失败并出现错误:
nc: connect to localhost port 3306 (tcp) failed: Connection refused
我用头撞墙试图找出原因,但无法深入了解所以希望这里有人可以提供帮助。
我在 https://github.com/jamiekt/democratising-dataproc 主持了 terraform 项目。重现问题非常简单,按照以下步骤操作:
- Install terraform 如果你还没有
- 安装
gcloud
如果您还没有安装 - 创建一个新的 GCP 项目
- 为您的新项目启用Cloud Dataproc API
gcloud auth application-default login #creates a file containing credentials that terraform will use
git clone git@github.com:jamiekt/democratising-dataproc.git && cd democratising-dataproc
export GCP_PROJECT=name-of-project-you-just-created
make init
make apply
应该成功启动一个网络、子网、云sql实例、几个存储桶(其中一个包含云-sql-proxy.sh)、一个服务帐户,然后防火墙在尝试创建 dataproc 集群时失败。
如果有人能看一看并告诉我为什么失败,我将不胜感激。
看来您使用的不是最新版本的cloud-sql-proxy.sh script in the cloud-sql-proxy.sh.tmpl template (diff).
您可能想尝试使用来自 Dataproc Cloud SQL I/O and Hive Metastore 初始化操作的最新脚本版本更新您的模板。
这里有一些问题现在已经解决了:
- service_account_scopes needed specifying and
hive:hive.metastore.warehouse.dir
property needed setting - Terraform removes the default root user from MySQL so it has to be recreated. I was missing (amongst other things) the host attribute (
host = '%'
) - 其他各种事情
state of the repo at the time of posting this message 将按预期工作(即使用 Terraform 创建一个使用共享配置单元元存储的数据处理集群)。
感谢@igor-dvorzhak 的回复,您对配置 Hive Metastore 以使用云的文章 link SQL 让我走上了正确的轨道..
按照 here
中的教程,我遇到了完全相同的问题我看不出你给出的修复是什么。 似乎配置单元尝试通过本地主机访问数据库我不明白为什么 这是来自 dataproc 的日志:
Synchronizing state of mysql.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable mysql
Removed /etc/systemd/system/multi-user.target.wants/mysql.service.
2020-10-07 14:05:49 URL:https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 [16361206/16361206] -> "cloud_sql_proxy.linux.amd64" [1]
Created symlink /etc/systemd/system/multi-user.target.wants/cloud-sql-proxy.service → /usr/lib/systemd/system/cloud-sql-proxy.service.
About to run 'nc -zv localhost 3306' with retries...
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 1 failed! Sleeping 1.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 2 failed! Sleeping 1.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 3 failed! Sleeping 2.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 4 failed! Sleeping 3.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 5 failed! Sleeping 5.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 6 failed! Sleeping 8.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 7 failed! Sleeping 13.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 8 failed! Sleeping 21.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 9 failed! Sleeping 34.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 10 failed! Sleeping 55.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 11 failed! Sleeping 89.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 12 failed! Sleeping 144.
Final attempt of 'nc -zv localhost 3306'...
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused