关闭和打开 Google 云虚拟机(计算引擎)后动态端口转发失败
Dynamic port forwarding fails after turning off and on Google Cloud virtual machine (compute engine)
我正在使用动态端口转发连接到我的 Spark 集群主节点,这样我就可以在我的本地机器上打开 jupyter 笔记本 Web 界面。
我按照 Google Cloud Dataproc 教程中的说明进行操作:https://cloud.google.com/dataproc/docs/tutorials/jupyter-notebook
我按照建议使用以下命令创建了 ssh 漏斗:
gcloud compute ssh --zone=<cluster-zone> --ssh-flag="-D" --ssh-flag="10000" --ssh-flag="-N" "<cluster-name>-m"
并打开网页界面:
<browser executable path> \
"http://<cluster-name>-m:8123" \
--proxy-server="socks5://localhost:10000" \
--host-resolver-rules="MAP * 0.0.0.0 , EXCLUDE localhost" \
--user-data-dir=/tmp/
我第一次尝试时效果非常好。
但是,一旦我关闭我的 goole 计算引擎并在一段时间后将其打开,完全相同的命令就不起作用,并给出以下错误消息:
debug1: Connection to port 10000 forwarding to socks port 0 requested.
debug2: fd 8 setting TCP_NODELAY
debug3: fd 8 is O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 2: new [dynamic-tcpip]
debug2: channel 2: pre_dynamic: have 0
debug2: channel 2: pre_dynamic: have 3
debug2: channel 2: decode socks5
debug2: channel 2: socks5 auth done
debug2: channel 2: pre_dynamic: need more
debug2: channel 2: pre_dynamic: have 0
debug2: channel 2: pre_dynamic: have 19
debug2: channel 2: decode socks5
debug2: channel 2: socks5 post auth
debug2: channel 2: dynamic request: socks5 host cluster-1-m port 8123 command 1
channel 2: open failed: connect failed: Connection refused
debug2: channel 2: zombie
debug2: channel 2: garbage collecting
debug1: channel 2: free: direct-tcpip: listening port 10000 for cluster-1-m port 8123, connect from ::1 port 49535 to ::1 port 10000, nchannels 3
debug3: channel 2: status: The following connections are open:
等待help:D
Jupyter notebook 内核在重启后不会重新启动。机器启动后,您需要自己手动重启笔记本,例如:
gcloud compute ssh <cluster-name>-m
nohup /usr/local/bin/miniconda/bin/jupyter notebook --no-browser > /var/log/jupyter_notebook.log 2>&1 &
内核启动后 运行,您应该可以通过代理访问网络 UI。
注意:一般情况下,Dataproc 不支持停止或重启整个集群。
我正在使用动态端口转发连接到我的 Spark 集群主节点,这样我就可以在我的本地机器上打开 jupyter 笔记本 Web 界面。
我按照 Google Cloud Dataproc 教程中的说明进行操作:https://cloud.google.com/dataproc/docs/tutorials/jupyter-notebook
我按照建议使用以下命令创建了 ssh 漏斗:
gcloud compute ssh --zone=<cluster-zone> --ssh-flag="-D" --ssh-flag="10000" --ssh-flag="-N" "<cluster-name>-m"
并打开网页界面:
<browser executable path> \
"http://<cluster-name>-m:8123" \
--proxy-server="socks5://localhost:10000" \
--host-resolver-rules="MAP * 0.0.0.0 , EXCLUDE localhost" \
--user-data-dir=/tmp/
我第一次尝试时效果非常好。
但是,一旦我关闭我的 goole 计算引擎并在一段时间后将其打开,完全相同的命令就不起作用,并给出以下错误消息:
debug1: Connection to port 10000 forwarding to socks port 0 requested.
debug2: fd 8 setting TCP_NODELAY
debug3: fd 8 is O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 2: new [dynamic-tcpip]
debug2: channel 2: pre_dynamic: have 0
debug2: channel 2: pre_dynamic: have 3
debug2: channel 2: decode socks5
debug2: channel 2: socks5 auth done
debug2: channel 2: pre_dynamic: need more
debug2: channel 2: pre_dynamic: have 0
debug2: channel 2: pre_dynamic: have 19
debug2: channel 2: decode socks5
debug2: channel 2: socks5 post auth
debug2: channel 2: dynamic request: socks5 host cluster-1-m port 8123 command 1
channel 2: open failed: connect failed: Connection refused
debug2: channel 2: zombie
debug2: channel 2: garbage collecting
debug1: channel 2: free: direct-tcpip: listening port 10000 for cluster-1-m port 8123, connect from ::1 port 49535 to ::1 port 10000, nchannels 3
debug3: channel 2: status: The following connections are open:
等待help:D
Jupyter notebook 内核在重启后不会重新启动。机器启动后,您需要自己手动重启笔记本,例如:
gcloud compute ssh <cluster-name>-m
nohup /usr/local/bin/miniconda/bin/jupyter notebook --no-browser > /var/log/jupyter_notebook.log 2>&1 &
内核启动后 运行,您应该可以通过代理访问网络 UI。
注意:一般情况下,Dataproc 不支持停止或重启整个集群。