通过 Linux tap 设备的 TCP 隧道

TCP tunneling through Linux tap device

我已经创建了一个 tap0 设备(IP 10.0.0.101),并且正在使用 zeromq 的 pgm pub/sub(例如 pgm://192.168.100.2;234.5.6.7:5555)从点击 zmq 订阅者,反之亦然。这个想法是使用 pgm 创建一个虚拟网络。我在网络上有 2 个 Tap 主机:10.0.0.101、10.0.0.11。他们在 192.168.106.126、192.168.106.55 也有物理以太网适配器。

'问题是 ping 有效,但 http 和 ssh 协议无效。

Wireshark 显示成功的 TCP 启动序列,但随后我开始看到重复的 ACK、重传,并且 curl 和 ssh 挂起一段时间并最终出错。

下面是 Wireshark 的一个片段,后面是大部分(希望如此)相关的 ruby 源代码。这是使用 rb_tuntap 和 ffi-rzmq 宝石。

No.     Time               Source                Destination           Protocol Length Info
      7 11:41:45.464867000 10.0.0.11             10.0.0.101            TCP      74     51659 > 3000 [SYN] Seq=0 Win=14600 Len=0 MSS=1460 SACK_PERM=1 TSval=1953042 TSecr=0 WS=64

Frame 7: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface 0
Ethernet II, Src: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f), Dst: 56:c8:52:17:31:67 (56:c8:52:17:31:67)
Internet Protocol Version 4, Src: 10.0.0.11 (10.0.0.11), Dst: 10.0.0.101 (10.0.0.101)
Transmission Control Protocol, Src Port: 51659 (51659), Dst Port: 3000 (3000), Seq: 0, Len: 0

No.     Time               Source                Destination           Protocol Length Info
      8 11:41:45.464956000 10.0.0.101            10.0.0.11             TCP      74     3000 > 51659 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=10191992 TSecr=1953042 WS=128

Frame 8: 74 bytes on wire (592 bits), 74 bytes captured (592 bits) on interface 0
Ethernet II, Src: 56:c8:52:17:31:67 (56:c8:52:17:31:67), Dst: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f)
Internet Protocol Version 4, Src: 10.0.0.101 (10.0.0.101), Dst: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 3000 (3000), Dst Port: 51659 (51659), Seq: 0, Ack: 1, Len: 0

No.     Time               Source                Destination           Protocol Length Info
     11 11:41:45.473101000 10.0.0.11             10.0.0.101            TCP      66     51659 > 3000 [ACK] Seq=1 Ack=1 Win=14656 Len=0 TSval=1953044 TSecr=10191992

Frame 11: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface 0
Ethernet II, Src: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f), Dst: 56:c8:52:17:31:67 (56:c8:52:17:31:67)
Internet Protocol Version 4, Src: 10.0.0.11 (10.0.0.11), Dst: 10.0.0.101 (10.0.0.101)
Transmission Control Protocol, Src Port: 51659 (51659), Dst Port: 3000 (3000), Seq: 1, Ack: 1, Len: 0

No.     Time               Source                Destination           Protocol Length Info
     12 11:41:45.473429000 10.0.0.11             10.0.0.101            HTTP     145    GET / HTTP/1.1 

Frame 12: 145 bytes on wire (1160 bits), 145 bytes captured (1160 bits) on interface 0
Ethernet II, Src: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f), Dst: 56:c8:52:17:31:67 (56:c8:52:17:31:67)
Internet Protocol Version 4, Src: 10.0.0.11 (10.0.0.11), Dst: 10.0.0.101 (10.0.0.101)
Transmission Control Protocol, Src Port: 51659 (51659), Dst Port: 3000 (3000), Seq: 1, Ack: 1, Len: 79
Hypertext Transfer Protocol

No.     Time               Source                Destination           Protocol Length Info
     13 11:41:45.473460000 10.0.0.101            10.0.0.11             TCP      66     3000 > 51659 [ACK] Seq=1 Ack=80 Win=29056 Len=0 TSval=10192001 TSecr=1953046

Frame 13: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface 0
Ethernet II, Src: 56:c8:52:17:31:67 (56:c8:52:17:31:67), Dst: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f)
Internet Protocol Version 4, Src: 10.0.0.101 (10.0.0.101), Dst: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 3000 (3000), Dst Port: 51659 (51659), Seq: 1, Ack: 80, Len: 0

No.     Time               Source                Destination           Protocol Length Info
     15 11:41:45.491555000 10.0.0.101            10.0.0.11             TCP      717    [TCP segment of a reassembled PDU]

Frame 15: 717 bytes on wire (5736 bits), 717 bytes captured (5736 bits) on interface 0
Ethernet II, Src: 56:c8:52:17:31:67 (56:c8:52:17:31:67), Dst: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f)
Internet Protocol Version 4, Src: 10.0.0.101 (10.0.0.101), Dst: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 3000 (3000), Dst Port: 51659 (51659), Seq: 1, Ack: 80, Len: 651

No.     Time               Source                Destination           Protocol Length Info
     16 11:41:45.491599000 10.0.0.101            10.0.0.11             TCP      1514   [TCP segment of a reassembled PDU]

Frame 16: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) on interface 0
Ethernet II, Src: 56:c8:52:17:31:67 (56:c8:52:17:31:67), Dst: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f)
Internet Protocol Version 4, Src: 10.0.0.101 (10.0.0.101), Dst: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 3000 (3000), Dst Port: 51659 (51659), Seq: 652, Ack: 80, Len: 1448

No.     Time               Source                Destination           Protocol Length Info
     21 11:41:45.496998000 10.0.0.11             10.0.0.101            TCP      66     51659 > 3000 [ACK] Seq=80 Ack=652 Win=17536 Len=0 TSval=1953058 TSecr=10192019

Frame 21: 66 bytes on wire (528 bits), 66 bytes captured (528 bits) on interface 0
Ethernet II, Src: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f), Dst: 56:c8:52:17:31:67 (56:c8:52:17:31:67)
Internet Protocol Version 4, Src: 10.0.0.11 (10.0.0.11), Dst: 10.0.0.101 (10.0.0.101)
Transmission Control Protocol, Src Port: 51659 (51659), Dst Port: 3000 (3000), Seq: 80, Ack: 652, Len: 0

No.     Time               Source                Destination           Protocol Length Info
     22 11:41:45.497026000 10.0.0.101            10.0.0.11             HTTP     231    HTTP/1.1 200 OK   (text/html)

Frame 22: 231 bytes on wire (1848 bits), 231 bytes captured (1848 bits) on interface 0
Ethernet II, Src: 56:c8:52:17:31:67 (56:c8:52:17:31:67), Dst: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f)
Internet Protocol Version 4, Src: 10.0.0.101 (10.0.0.101), Dst: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 3000 (3000), Dst Port: 51659 (51659), Seq: 2100, Ack: 80, Len: 165
[3 Reassembled TCP Segments (2264 bytes): #15(651), #16(1448), #22(165)]
Hypertext Transfer Protocol
Line-based text data: text/html

No.     Time               Source                Destination           Protocol Length Info
     25 11:41:45.502736000 10.0.0.11             10.0.0.101            TCP      78     [TCP Dup ACK 21#1] 51659 > 3000 [ACK] Seq=80 Ack=652 Win=17536 Len=0 TSval=1953061 TSecr=10192019 SLE=2100 SRE=2265

Frame 25: 78 bytes on wire (624 bits), 78 bytes captured (624 bits) on interface 0
Ethernet II, Src: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f), Dst: 56:c8:52:17:31:67 (56:c8:52:17:31:67)
Internet Protocol Version 4, Src: 10.0.0.11 (10.0.0.11), Dst: 10.0.0.101 (10.0.0.101)
Transmission Control Protocol, Src Port: 51659 (51659), Dst Port: 3000 (3000), Seq: 80, Ack: 652, Len: 0

No.     Time               Source                Destination           Protocol Length Info
     26 11:41:45.504245000 10.0.0.101            10.0.0.11             TCP      1514   [TCP Retransmission] 3000 > 51659 [ACK] Seq=652 Ack=80 Win=29056 Len=1448 TSval=10192032 TSecr=1953061[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]

Frame 26: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) on interface 0
Ethernet II, Src: 56:c8:52:17:31:67 (56:c8:52:17:31:67), Dst: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f)
Internet Protocol Version 4, Src: 10.0.0.101 (10.0.0.101), Dst: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 3000 (3000), Dst Port: 51659 (51659), Seq: 652, Ack: 80, Len: 1448
[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]

No.     Time               Source                Destination           Protocol Length Info
     33 11:41:45.711324000 10.0.0.101            10.0.0.11             TCP      1514   [TCP Retransmission] 3000 > 51659 [ACK] Seq=652 Ack=80 Win=29056 Len=1448 TSval=10192239 TSecr=1953061[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]

Frame 33: 1514 bytes on wire (12112 bits), 1514 bytes captured (12112 bits) on interface 0
Ethernet II, Src: 56:c8:52:17:31:67 (56:c8:52:17:31:67), Dst: 3a:e2:d5:f3:8e:6f (3a:e2:d5:f3:8e:6f)
Internet Protocol Version 4, Src: 10.0.0.101 (10.0.0.101), Dst: 10.0.0.11 (10.0.0.11)
Transmission Control Protocol, Src Port: 3000 (3000), Dst Port: 51659 (51659), Seq: 652, Ack: 80, Len: 1448
[Reassembly error, protocol TCP: New fragment overlaps old data (retransmission?)]

Ruby代码:

def run

    stop = false

    Signal.trap("SIGINT")  {
        stop=true
    }

    tap.up

    #binding.pry
    pids = []
    begin

        # tap => zmq_pub 
        pids << Thread.new do
            while !stop do
                read_and_pub
            end
        end

        # zmq_sub => tap
        pids << Thread.new do
            while !stop do
                sub_and_write
            end
        end

    rescue
        stop=true
    end

    pids.each { |pid| pid.join }

    tap.down
    tap.close
    self.tap = nil

    zmq_pub.close
    zmq_sub.close
    zmq_ctx.terminate

end # run

def read_and_pub

    selected = IO.select([tap.to_io],nil,nil,1)
    if !selected.nil? and !selected[0].nil? and selected[0].length>0 then
        msg = tap.to_io.sysread(tap.mtu)
        if !msg.nil? && msg.length > 0 then
            sent = zmq_pub.send_string(msg) 
            if sent != msg.length
                puts "**** published #{sent}/#{msg.length} ****"
            end
            print_packet msg, tap
        else
            puts "IO.select returned #{selected} but msg is #{msg}"
        end
    end
end

def sub_and_write
    msg = ''
    zmq_sub.recv_string msg
    if !msg.nil? && msg.length > 0 then
        sent = tap.to_io.syswrite(msg)
        if sent != msg.length
            puts "**** wrote #{sent}/#{msg.length} ****"
        end
        print_packet msg, zmq_sub
    end
end

问题在这里:tap.to_io.sysread(tap.mtu)。似乎 tap.mtu 不一定是 sysread 将接收的最大字节数,未读字节似乎从 tap 接口中被丢弃。我将代码更改为 tap.to_io.sysread(10000) 并且 http 和 ssh 按预期工作。我收到的最大帧是1514