cloud-sql-GCP Dataproc 代理失败,nc:连接到本地主机端口 3306 (tcp) 失败:连接被拒绝

cloud-sql-proxy for GCP Dataproc fails, nc: connect to localhost port 3306 (tcp) failed: Connection refused

我正在使用 Terraform 创建一个数据处理集群,该集群使用 GCP 云sql 实例作为 hivemetastore,terrafrm 项目创建集群及其所有先决条件(网络、服务帐户、云sql 实例和用户等)。
提供 cloud-sql-proxy.sh 来帮助解决这个问题,但是我无法让它工作,当集群创建时 cloud-sql-proxy.sh 失败并出现错误:

nc: connect to localhost port 3306 (tcp) failed: Connection refused

我用头撞墙试图找出原因,但无法深入了解所以希望这里有人可以提供帮助。

我在 https://github.com/jamiekt/democratising-dataproc 主持了 terraform 项目。重现问题非常简单,按照以下步骤操作:

  1. Install terraform 如果你还没有
  2. 安装 gcloud 如果您还没有安装
  3. 创建一个新的 GCP 项目
  4. 为您的新项目启用Cloud Dataproc API
  5. gcloud auth application-default login #creates a file containing credentials that terraform will use
  6. git clone git@github.com:jamiekt/democratising-dataproc.git && cd democratising-dataproc
  7. export GCP_PROJECT=name-of-project-you-just-created
  8. make init
  9. make apply

应该成功启动一个网络、子网、云sql实例、几个存储桶(其中一个包含云-sql-proxy.sh)、一个服务帐户,然后防火墙在尝试创建 dataproc 集群时失败。

如果有人能看一看并告诉我为什么失败,我将不胜感激。

看来您使用的不是最新版本的cloud-sql-proxy.sh script in the cloud-sql-proxy.sh.tmpl template (diff).

您可能想尝试使用来自 Dataproc Cloud SQL I/O and Hive Metastore 初始化操作的最新脚本版本更新您的模板。

这里有一些问题现在已经解决了:

state of the repo at the time of posting this message 将按预期工作(即使用 Terraform 创建一个使用共享配置单元元存储的数据处理集群)。

感谢@igor-dvorzhak 的回复,您对配置 Hive Metastore 以使用云的文章 link SQL 让我走上了正确的轨道..

按照 here

中的教程,我遇到了完全相同的问题

我看不出你给出的修复是什么。 似乎配置单元尝试通过本地主机访问数据库我不明白为什么 这是来自 dataproc 的日志:

Synchronizing state of mysql.service with SysV service script with /lib/systemd/systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install disable mysql
Removed /etc/systemd/system/multi-user.target.wants/mysql.service.
2020-10-07 14:05:49 URL:https://dl.google.com/cloudsql/cloud_sql_proxy.linux.amd64 [16361206/16361206] -> "cloud_sql_proxy.linux.amd64" [1]
Created symlink /etc/systemd/system/multi-user.target.wants/cloud-sql-proxy.service → /usr/lib/systemd/system/cloud-sql-proxy.service.
About to run 'nc -zv localhost 3306' with retries...
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 1 failed! Sleeping 1.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 2 failed! Sleeping 1.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 3 failed! Sleeping 2.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 4 failed! Sleeping 3.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 5 failed! Sleeping 5.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 6 failed! Sleeping 8.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 7 failed! Sleeping 13.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 8 failed! Sleeping 21.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 9 failed! Sleeping 34.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 10 failed! Sleeping 55.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 11 failed! Sleeping 89.
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused
'nc -zv localhost 3306' attempt 12 failed! Sleeping 144.
Final attempt of 'nc -zv localhost 3306'...
nc: connect to localhost port 3306 (tcp) failed: Connection refused
nc: connect to localhost port 3306 (tcp) failed: Connection refused