Nginx proxy_bind 失败
Nginx proxy_bind failing
我正在尝试将 nginx 配置为使用监听和 proxy_bind 指令。
# /opt/sbin/nginx -v
nginx version: nginx/1.19.2 (x86_64-pc-linux-gnu)
# ifconfig br0:0 192.168.0.2
# opkg install nginx-ssl
# opkg install ca-bundle
# cat /opt/etc/nginx/nginx.conf
user admin root;
#user nobody;
worker_processes 1;
events {
worker_connections 64;
}
http {
# HTTPS server
server {
listen 192.168.0.2:443 ssl;
server_name $host;
ssl_certificate /etc/cert.pem;
ssl_certificate_key /etc/key.pem;
ssl_client_certificate /opt/etc/ssl/cert.pem;
ssl_verify_client optional;
ssl_verify_depth 2;
proxy_ssl_server_name on;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
# root html;
# index index.html index.htm;
resolver 103.86.99.100;
# proxy_bind 192.168.0.2:12345;
proxy_bind $server_addr:12345;
# proxy_bind $remote_addr:12345 transparent;
proxy_pass $scheme://$host;
}
}
}
我试过更改user admin root;，这是该路由器的root 用户。我试过使用 proxy_bind 192.168.0.2;、proxy_bind 192.168.0.2 transparent;、proxy_bind $server_addr; 和 proxy_bind $server_addr transparent; 的不同组合。 None 在使用 tcpdump 验证时，它们似乎有效。 nginx 始终使用外部 WAN 地址 (100.64.8.236)。
Ifconfig 输出：
# ifconfig
br0 Link encap:Ethernet HWaddr C0:56:27:D1:B8:A4
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
RX packets:10243803 errors:0 dropped:0 overruns:0 frame:0
TX packets:5440860 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:14614392834 (13.6 GiB) TX bytes:860977246 (821.0 MiB)
br0:0 Link encap:Ethernet HWaddr C0:56:27:D1:B8:A4
inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
vlan2 Link encap:Ethernet HWaddr C0:56:27:D1:B8:A4
inet addr:100.64.8.236 Bcast:100.64.15.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1757588 errors:0 dropped:0 overruns:0 frame:0
TX packets:613625 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2267961441 (2.1 GiB) TX bytes:139435610 (132.9 MiB)
路由输出：
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.10.0.17 * 255.255.255.255 UH 0 0 0 tun12
89.38.98.142 100.64.8.1 255.255.255.255 UGH 0 0 0 vlan2
100.64.8.1 * 255.255.255.255 UH 0 0 0 vlan2
10.15.0.65 * 255.255.255.255 UH 0 0 0 tun11
192.168.2.1 * 255.255.255.255 UH 0 0 0 vlan3
51.68.180.4 100.64.8.1 255.255.255.255 UGH 0 0 0 vlan2
192.168.2.0 * 255.255.255.0 U 0 0 0 vlan3
192.168.0.0 * 255.255.255.0 U 0 0 0 br0
100.64.8.0 * 255.255.248.0 U 0 0 0 vlan2
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 100.64.8.1 0.0.0.0 UG 0 0 0 vlan2
Tcpdump 输出：
Client Remote_Addr (192.168.0.154:$port) == Request => Nginx Reverse Proxy Server - Listener (192.168.0.2:443)
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
Nginx 反向代理服务器 - 侦听器 (192.168.0.2:443) == 响应 => 客户端 Remote_Addr (192.168.0.154:$port)
07:19:06.841377 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
07:19:06.841411 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
Nginx 反向代理服务器 - 发送方 (100.64.8.236:12345) == 请求 => 上游目标服务器 - 监听方 (104.27.161.206:443)
07:19:11.885314 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: 100.64.8.236.12345 > 104.27.161.206.443: Flags [S], seq 3472185855, win 5840, options [mss 1460,sackOK,TS val 331214 ecr 0,nop,wscale 4], length 0
上游目标服务器 - 监听器 (104.27.161.206:443) == 响应 => Nginx 反向代理服务器 - 发送器 (100.64.8.236:12345)
07:19:11.887683 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 68: 104.27.161.206.443 > 100.64.8.236.12345: Flags [S.], seq 2113436779, ack 3472185856, win 65535, options [mss 1400,nop,nop,sackOK,nop,wscale 10], length 0
注意：Nginx反向代理服务器（监听器）和Nginx反向代理服务器（发送器）MAC地址是同一块硬件
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
07:19:06.841377 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
07:19:06.841411 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
07:19:11.885314 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: 100.64.8.236.12345 > 104.27.161.206.443: Flags [S], seq 3472185855, win 5840, options [mss 1460,sackOK,TS val 331214 ecr 0,nop,wscale 4], length 0
07:19:11.887683 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 68: 104.27.161.206.443 > 100.64.8.236.12345: Flags [S.], seq 2113436779, ack 3472185856, win 65535, options [mss 1400,nop,nop,sackOK,nop,wscale 10], length 0
07:19:11.887948 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 100.64.8.236.12345 > 104.27.161.206.443: Flags [.], ack 1, win 365, length 0
07:19:11.888854 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 264: 100.64.8.236.12345 > 104.27.161.206.443: Flags [P.], seq 1:209, ack 1, win 365, length 208
07:19:11.890844 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 62: 104.27.161.206.443 > 100.64.8.236.12345: Flags [.], ack 209, win 66, length 0
07:19:11.893154 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 1516: 104.27.161.206.443 > 100.64.8.236.12345: Flags [.], seq 1:1461, ack 209, win 66, length 1460
07:19:11.893316 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 100.64.8.236.12345 > 104.27.161.206.443: Flags [.], ack 1461, win 548, length 0
07:19:11.893161 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 1000: 104.27.161.206.443 > 100.64.8.236.12345: Flags [P.], seq 1461:2405, ack 209, win 66, length 944
Iptables 输出：
# iptables -t mangle -I PREROUTING -i vlan2 -p tcp -m multiport --dport 12345 -j MARK --set-mark 0x2000/0x2000
# iptables -t mangle -I POSTROUTING -o vlan2 -p tcp -m multiport --sport 12345 -j MARK --set-mark 0x8000/0x8000
注意：数据包正在匹配并被标记，但没有被路由到适当的接口。我在想可能为时已晚。
# iptables -t mangle -L -v -n
Chain PREROUTING (policy ACCEPT 5506K packets, 8051M bytes)
pkts bytes target prot opt in out source destination
33 15329 MARK tcp -- vlan2 * 0.0.0.0/0 0.0.0.0/0 multiport dports 12345 MARK or 0x2000
Chain POSTROUTING (policy ACCEPT 2832K packets, 171M bytes)
pkts bytes target prot opt in out source destination
30 4548 MARK tcp -- * vlan2 0.0.0.0/0 0.0.0.0/0 multiport sports 12345 MARK or 0x8000
反向代理请求到达目的地并返回，但使用外部 WAN 地址 (100.64.8.236:12345) 而不是内部主机地址 (192.168.0.2:12345)。
编辑：我发现在启动流量时从 192.168.0.2:12345（nginx worker）发送了一个 SYN 数据包。没有更多的回应。我想知道 192.168.0.2:12345 是否没有收到 SYN,ACK 是使用下一个可用接口地址（即 100.64.8.236:12345）的 nginx worker？
# netstat -anp|grep 12345
tcp 0 1 192.168.0.2:12345 172.64.163.36:443 SYN_SENT 14176/nginx: worker
我的目标是使用 iptables 标记 192.168.0.2:12345 数据包，以便通过已建立的 OpenVPN 隧道 (tun12) 进行路由。
proxy_bind 指令似乎失败了。
有什么想法吗？
全部：
查看 iptables 链工作流程后，我发现 Nginx Worker (192.168.0.2:12345) 外部接口与 OUTPUT 链关联。
(192.168.0.2:12345) OUTPUT ==>
(192.168.0.154:$port) PREROUTING ==> (100.64.8.236:12345) POSTROUTING ==>
Windows Client (192.168.0.154:$port) ==> Nginx Master (192.168.0.2:443) | Nginx Worker (100.64.8.236:12345) ==> Upstream Desination Server (104.27.161.206:443)
<== POSTROUTING (192.168.0.2:443) <== PREROUTING (104.27.161.206:443)
一旦添加了适当的 iptables OUTPUT 规则，使用正确的接口 (vlan2)，离开 Nginx Worker (100.64.8.236:12345) 的数据包就会被适当地标记并路由到 OpenVPN 隧道。
# iptables -t mangle -I OUTPUT -o vlan2 -p tcp -m multiport --sport 12345 -j MARK --set-mark 0x2000/0x2000
现在，我只需要弄清楚 Nginx SSL 客户端 CA 信任配置，我们就可以开始工作了。
希望这对以后的人有所帮助。
恭敬地，
加里
我正在尝试将 nginx 配置为使用监听和 proxy_bind 指令。
# /opt/sbin/nginx -v
nginx version: nginx/1.19.2 (x86_64-pc-linux-gnu)
# ifconfig br0:0 192.168.0.2
# opkg install nginx-ssl
# opkg install ca-bundle
# cat /opt/etc/nginx/nginx.conf
user admin root;
#user nobody;
worker_processes 1;
events {
worker_connections 64;
}
http {
# HTTPS server
server {
listen 192.168.0.2:443 ssl;
server_name $host;
ssl_certificate /etc/cert.pem;
ssl_certificate_key /etc/key.pem;
ssl_client_certificate /opt/etc/ssl/cert.pem;
ssl_verify_client optional;
ssl_verify_depth 2;
proxy_ssl_server_name on;
ssl_session_cache shared:SSL:1m;
ssl_session_timeout 5m;
ssl_ciphers HIGH:!aNULL:!MD5;
ssl_prefer_server_ciphers on;
location / {
# root html;
# index index.html index.htm;
resolver 103.86.99.100;
# proxy_bind 192.168.0.2:12345;
proxy_bind $server_addr:12345;
# proxy_bind $remote_addr:12345 transparent;
proxy_pass $scheme://$host;
}
}
}
我试过更改user admin root;，这是该路由器的root 用户。我试过使用 proxy_bind 192.168.0.2;、proxy_bind 192.168.0.2 transparent;、proxy_bind $server_addr; 和 proxy_bind $server_addr transparent; 的不同组合。 None 在使用 tcpdump 验证时，它们似乎有效。 nginx 始终使用外部 WAN 地址 (100.64.8.236)。
Ifconfig 输出：
# ifconfig
br0 Link encap:Ethernet HWaddr C0:56:27:D1:B8:A4
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
RX packets:10243803 errors:0 dropped:0 overruns:0 frame:0
TX packets:5440860 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:14614392834 (13.6 GiB) TX bytes:860977246 (821.0 MiB)
br0:0 Link encap:Ethernet HWaddr C0:56:27:D1:B8:A4
inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING ALLMULTI MULTICAST MTU:1500 Metric:1
vlan2 Link encap:Ethernet HWaddr C0:56:27:D1:B8:A4
inet addr:100.64.8.236 Bcast:100.64.15.255 Mask:255.255.248.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1757588 errors:0 dropped:0 overruns:0 frame:0
TX packets:613625 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:2267961441 (2.1 GiB) TX bytes:139435610 (132.9 MiB)
路由输出：
# route
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
10.10.0.17 * 255.255.255.255 UH 0 0 0 tun12
89.38.98.142 100.64.8.1 255.255.255.255 UGH 0 0 0 vlan2
100.64.8.1 * 255.255.255.255 UH 0 0 0 vlan2
10.15.0.65 * 255.255.255.255 UH 0 0 0 tun11
192.168.2.1 * 255.255.255.255 UH 0 0 0 vlan3
51.68.180.4 100.64.8.1 255.255.255.255 UGH 0 0 0 vlan2
192.168.2.0 * 255.255.255.0 U 0 0 0 vlan3
192.168.0.0 * 255.255.255.0 U 0 0 0 br0
100.64.8.0 * 255.255.248.0 U 0 0 0 vlan2
127.0.0.0 * 255.0.0.0 U 0 0 0 lo
default 100.64.8.1 0.0.0.0 UG 0 0 0 vlan2
Tcpdump 输出：
Client Remote_Addr (192.168.0.154:$port) == Request => Nginx Reverse Proxy Server - Listener (192.168.0.2:443)
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
Nginx 反向代理服务器 - 侦听器 (192.168.0.2:443) == 响应 => 客户端 Remote_Addr (192.168.0.154:$port)
07:19:06.841377 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
07:19:06.841411 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
Nginx 反向代理服务器 - 发送方 (100.64.8.236:12345) == 请求 => 上游目标服务器 - 监听方 (104.27.161.206:443)
07:19:11.885314 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: 100.64.8.236.12345 > 104.27.161.206.443: Flags [S], seq 3472185855, win 5840, options [mss 1460,sackOK,TS val 331214 ecr 0,nop,wscale 4], length 0
上游目标服务器 - 监听器 (104.27.161.206:443) == 响应 => Nginx 反向代理服务器 - 发送器 (100.64.8.236:12345)
07:19:11.887683 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 68: 104.27.161.206.443 > 100.64.8.236.12345: Flags [S.], seq 2113436779, ack 3472185856, win 65535, options [mss 1400,nop,nop,sackOK,nop,wscale 10], length 0
注意：Nginx反向代理服务器（监听器）和Nginx反向代理服务器（发送器）MAC地址是同一块硬件
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
07:19:06.840468 In c8:1f:66:13:a1:11 ethertype IPv4 (0x0800), length 62: 192.168.0.154.55138 > 192.168.0.2.443: Flags [.], ack 1582, win 8212, length 0
07:19:06.841377 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
07:19:06.841411 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 192.168.0.2.443 > 192.168.0.154.55138: Flags [.], ack 1475, win 541, length 0
07:19:11.885314 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 76: 100.64.8.236.12345 > 104.27.161.206.443: Flags [S], seq 3472185855, win 5840, options [mss 1460,sackOK,TS val 331214 ecr 0,nop,wscale 4], length 0
07:19:11.887683 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 68: 104.27.161.206.443 > 100.64.8.236.12345: Flags [S.], seq 2113436779, ack 3472185856, win 65535, options [mss 1400,nop,nop,sackOK,nop,wscale 10], length 0
07:19:11.887948 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 100.64.8.236.12345 > 104.27.161.206.443: Flags [.], ack 1, win 365, length 0
07:19:11.888854 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 264: 100.64.8.236.12345 > 104.27.161.206.443: Flags [P.], seq 1:209, ack 1, win 365, length 208
07:19:11.890844 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 62: 104.27.161.206.443 > 100.64.8.236.12345: Flags [.], ack 209, win 66, length 0
07:19:11.893154 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 1516: 104.27.161.206.443 > 100.64.8.236.12345: Flags [.], seq 1:1461, ack 209, win 66, length 1460
07:19:11.893316 Out c0:56:27:d1:b8:a4 ethertype IPv4 (0x0800), length 56: 100.64.8.236.12345 > 104.27.161.206.443: Flags [.], ack 1461, win 548, length 0
07:19:11.893161 In 02:1f:a0:00:00:09 ethertype IPv4 (0x0800), length 1000: 104.27.161.206.443 > 100.64.8.236.12345: Flags [P.], seq 1461:2405, ack 209, win 66, length 944
Iptables 输出：
# iptables -t mangle -I PREROUTING -i vlan2 -p tcp -m multiport --dport 12345 -j MARK --set-mark 0x2000/0x2000
# iptables -t mangle -I POSTROUTING -o vlan2 -p tcp -m multiport --sport 12345 -j MARK --set-mark 0x8000/0x8000
注意：数据包正在匹配并被标记，但没有被路由到适当的接口。我在想可能为时已晚。
# iptables -t mangle -L -v -n
Chain PREROUTING (policy ACCEPT 5506K packets, 8051M bytes)
pkts bytes target prot opt in out source destination
33 15329 MARK tcp -- vlan2 * 0.0.0.0/0 0.0.0.0/0 multiport dports 12345 MARK or 0x2000
Chain POSTROUTING (policy ACCEPT 2832K packets, 171M bytes)
pkts bytes target prot opt in out source destination
30 4548 MARK tcp -- * vlan2 0.0.0.0/0 0.0.0.0/0 multiport sports 12345 MARK or 0x8000
反向代理请求到达目的地并返回，但使用外部 WAN 地址 (100.64.8.236:12345) 而不是内部主机地址 (192.168.0.2:12345)。
编辑：我发现在启动流量时从 192.168.0.2:12345（nginx worker）发送了一个 SYN 数据包。没有更多的回应。我想知道 192.168.0.2:12345 是否没有收到 SYN,ACK 是使用下一个可用接口地址（即 100.64.8.236:12345）的 nginx worker？
# netstat -anp|grep 12345
tcp 0 1 192.168.0.2:12345 172.64.163.36:443 SYN_SENT 14176/nginx: worker
我的目标是使用 iptables 标记 192.168.0.2:12345 数据包，以便通过已建立的 OpenVPN 隧道 (tun12) 进行路由。
proxy_bind 指令似乎失败了。
有什么想法吗？
全部：
查看 iptables 链工作流程后，我发现 Nginx Worker (192.168.0.2:12345) 外部接口与 OUTPUT 链关联。
(192.168.0.2:12345) OUTPUT ==>
(192.168.0.154:$port) PREROUTING ==> (100.64.8.236:12345) POSTROUTING ==>
Windows Client (192.168.0.154:$port) ==> Nginx Master (192.168.0.2:443) | Nginx Worker (100.64.8.236:12345) ==> Upstream Desination Server (104.27.161.206:443)
<== POSTROUTING (192.168.0.2:443) <== PREROUTING (104.27.161.206:443)
一旦添加了适当的 iptables OUTPUT 规则，使用正确的接口 (vlan2)，离开 Nginx Worker (100.64.8.236:12345) 的数据包就会被适当地标记并路由到 OpenVPN 隧道。
# iptables -t mangle -I OUTPUT -o vlan2 -p tcp -m multiport --sport 12345 -j MARK --set-mark 0x2000/0x2000
现在，我只需要弄清楚 Nginx SSL 客户端 CA 信任配置，我们就可以开始工作了。
希望这对以后的人有所帮助。
恭敬地，
加里