无法使用 IOCP 正确接受新连接 - 套接字句柄无效

Unable to accept a new connection properly using IOCP - Invalid socket handle

我正在学习 IOCP,并决定根据以下文章编写自己的包装器 class:
http://www.codeproject.com/Articles/13382/A-simple-application-using-I-O-Completion-Ports-an

我的项目是一个使用 IOCP 的 C++ TCP 服务器。客户端使用 send() 和 recv() 来发送和接收我无法更改的数据(据我所知,这不会造成任何问题,但我提到它以防万一)。它还使用 socket()(而不是 WSASocket())创建套接字。 一切似乎都工作正常(CreateIoCompletionPort 没有错误,我可以将套接字描述符添加到现有完成端口而没有任何错误。我通过在每个这些函数之后添加对 WSAGetLastError() 的调用来检查所有内容)。

(首先,请不要介意不一致的编码风格。我喜欢先让东西工作,然后再清理它。)

socket_ = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED);
setsockopt(socket_, IPPROTO_IP, SO_DEBUG | TCP_NODELAY, sockopt, 4);
ioctlsocket(socket_, FIONBIO, &ulSockMode_);
sin_.sin_family = AF_INET;
sin_.sin_port = htons((uint16_t)uiPort_));
hAccept_[0] = WSACreateEvent();  //only 1 event, I'm using an array for convenience

if (hAccept_ == WSA_INVALID_EVENT)
{
    //this is never executed
}
WSAEventSelect(socket_, hAccept_[0], FD_ACCEPT);

检测到传入连接后(我使用 WSAWaitForMultipleEevents 和 WSAEnumNetworkEvents 工作不会触发任何错误),我使用以下代码接受客户端(这就是问题开始的地方):

SOCKET sock_client{ INVALID_SOCKET };
int32_t len_si{ sizeof(SOCKADDR_IN) };
//sock_client = accept(socket_, reinterpret_cast<SOCKADDR*>(pSockAddr), &len_si); // this works fine
//sock_client = sock_client = WSAAccept(socket_, reinterpret_cast<SOCKADDR*>(pSockAddr), &len_si, NULL, 0);//works fine too
char buf[2 * (sizeof(SOCKADDR_IN) + 16)];
WSAOVERLAPPED wsaovl;
uint32_t bytes{ 0 };
BOOL b = AcceptEx(socket_, sock_client, (PVOID)buf, 0, sizeof(SOCKADDR_IN) + 16, sizeof(SOCKADDR_IN) + 16, reinterpret_cast<LPDWORD>(&bytes), &wsaovl); //this fails, returns 0
int32_t test = WSAGetLastError(); // this returns 6 (WSA_INVALID_HANDLE)

我不知道为什么它适用于 accept() 和 WSAAccept(),但它不适用于 AcceptEx()。

如果我使用 accept(),在接受客户端后我需要立即调用 WSARecv()。我还没有将任何东西发送回客户端,但我读到它需要在工作线程中的 GetQueuedCompletionStatus() 之前调用:

WSABUF* buf = new WSABUF;
OVERLAPPED* ovl = new OVERLAPPED;
int32_t flags{ 0 };
int32_t bytes{ 0 };
int32_t bytes_recv = WSARecv(client_socket, buf, 1, &flags, &bytes, ovl, NULL); // this returns -1
int32_t err = WSAGetLastError(); // this returns 6 (WSA_INVALID_HANDLE)

由于这不起作用,我的工作线程中的 GetQueuedCompletionStatus() 例程一直挂起(或者至少,我认为这是原因)

我这样做有什么问题吗?从昨天晚上开始,我一直在尝试四处搜索并修复它,我知道时间不多,但我真的看不出我做错了什么。

更新: 我已经更改了为 AcceptEx() 初始化套接字的方式。

SOCKET sock_client = WSASocket(AF_INET, SOCK_STREAM, IPPROTO_IP, NULL, 0, WSA_FLAG_OVERLAPPED);

 WSAOVERLAPPED wsaovl = {};

AcceptEx() 仍然 returns 错误,但是 WSAGetLastError() 返回的错误现在是 997 (WSA_IO_PENDING)。我不确定 I/O 操作到底是什么,我将如何修复它。

我在学习 I/O Completion Ports (IOCP) 时偶然发现了类似的障碍...

我认为问题在于,在IOCP套接字模型的方案中,最复杂的部分是'socket acceptance'的开始阶段。这就是为什么大多数教程都会跳过它并开始讨论如何处理 send/recv。

如果您想对 IOCP 有足够的了解,以便您可以实现生产软件,那么我对您的建议是研究它直到您完全掌握它(下面这个答案是不够的)。我推荐的文档是“Microsoft 网络编程 Windows - 第二版 ”的第 5 章。这本书可能很旧,但对 IOCP 有效。此外,文章“Windows via C/C++: Synchronous and Asynchronous Device I/O”涉及 IOCP 的某些方面,但没有足够的信息来制作生产软件。

我会尽力解释,但是,我必须警告你,这可能还不够。开始了...

因此,您缺少的部分是 "How to do 'socket acceptance' in an IOCP socket model"。

首先,让我们检查服务器上典型的 Winsock(非 IOCP)调用序列;

// (1) Create listen socket on server.
WSASocket()            

// (2) Bind an address to your listen socket.
bind()                 

// (3) Associate the listen socket with an event object on FD_ACCEPT event.
WSAEventSelect(,, FD_ACCEPT )       

// (4) Put socket in listen state - now, Windows listening for new 
//     connection requests. If new requests comes, the associated 
//     event object will be set.
listen()               

// (5) Wait on the event object associated on listen socket. This 
//     will get signaled when a new connection request comes.
WaitForSingleObject() {

     // (6) A network activity has occurred. Verify that FD_ACCEPT has 
     //     raised the event object. This also resets the event object
     //     so WaitForSingleObject() does not loop non-stop.
     WSAEnumNetworkEvents()  


     // (7) Understanding this part is important. The WSAAccept() doesn't
     //     just accept connection, it first creates a new socket and
     //     then associates it with the newly accepted connection.
     WSAAccept()
}

步骤 (7) 适用于非基于 IOCP 的模型。然而,从性能的角度来看——套接字创建是昂贵的。而且它会减慢连接接受过程。

在 IOCP 模型中,套接字是为新传入的连接请求预先创建的。不仅套接字是预先创建的,它们甚至在连接请求到来之前就与侦听套接字相关联。为此,Microsoft 提供了 扩展函数 。 IOCP 模型需要的两个这样的函数是 AcceptEx() & GetAcceptExSockaddrs().

注意:使用这些扩展函数时,需要在运行时加载它们以避免性能损失。这可以使用 WSAIoctl() 来实现。如需进一步阅读,请参阅有关 AcceptEx() 的 MSDN 文档。

警告: AcceptEx() 可用于设置新套接字以在连接接受过程中接收一些数据。需要禁用此功能,因为它使应用程序容易受到 DoS 攻击,即发出连接请求但未发送数据。接收应用程序将无限期地等待该套接字。为避免这种情况,只需为其 'dwReceiveDataLength' 参数传递 0 值即可。

如何为 IOCP 模型设置连接接受代码?

一种方法是;

// (1) Create IO completion port
CreateIoCompletionPort()

// (2) Have a method that creates worker threads say 'CreateWorkerThreads()'. 
//     This assign same method (say WorkerThread_Func()) to all worker threads. 
//     In the WorkerThread_Func() threads are blocked on call to 
//     GetQueuedCompletionStatus().
CreateWorkerThreads()

// (3) Create listen socket.
WSASocket()            

// (4) Associate listen socket to IO Completion Port created earlier.
CreateIoCompletionPort()

// (5) Bind an address to your listen socket.
bind()                 

// (6) Put socket in listen state - now, Windows listening for new 
//     connection requests. If a new request comes, GetQueuedCompletionStatus() 
//     will release a thread.
listen() 

// (7) Create sockets in advance and call AcceptEx on each of 
//     these sockets. If a new connection requests comes 
//     Windows will pick one of these sockets and associate the 
//     connection with it.
//
//     As an example, below loop will create 1000 sockets.


GUID GuidAcceptEx = WSAID_ACCEPTEX;
DWORD dwBytes;
LPFN_ACCEPTEX lpfnAcceptEx;

// First, load extension method.
int retCode = WSAIoctl(listenSocket,
                        SIO_GET_EXTENSION_FUNCTION_POINTER,
                        &GuidAcceptEx,
                        sizeof(GuidAcceptEx),
                        &lpfnAcceptEx,
                        sizeof(lpfnAcceptEx),
                        &dwBytes,
                        NULL,
                        NULL
                        );

for( /* loop for 1000 times */ ) {
    SOCKET preemptiveSocket = WSASocket(, , , , , WSA_FLAG_OVERLAPPED);
    lpfnAcceptEx(listenSocket, preemptiveSocket,,,,,,);
}

这实质上让您的应用程序准备好以 IOCP 方式接受套接字。当新的连接请求到来时,正在等待 GetQueuedCompletionStatus() 的工作线程之一将被释放并移交指向 数据结构 的指针。这将具有由 lpfnAcceptEx() 创建的套接字。 流程是否完整?还没有。通过 AcceptEx() 调用接受的套接字不继承 listenSocket 的属性。为此,您需要致电;

  setsockopt( acceptSocket, SOL_SOCKET, SO_UPDATE_ACCEPT_CONTEXT, 
                          (char*)&listenSocket, sizeof(listenSocket) );

现在,acceptSocket 很适合用于 WSASend / WSARecv!

少了点东西!我跳过了有关工作线程如何从 GetQueuedCompletionStatus() 获取 acceptedSocket 的部分? 答案是,将您特制的结构传递给 lpfnAcceptEx()。当 GetQueuedCompletionStatus() returns 时,它将具有包含您将传递的套接字的数据结构。 如何制作这样的结构?通过创建一个以 'WSAOVERLAPPED' 作为第一个成员的结构。在第一个成员之后,您可以拥有自己的任何成员。例如,我的结构看起来像;

typedef struct _WSAOVERLAPPEDPLUS
{
    WSAOVERLAPPED ProviderOverlapped; // 'WSAOVERLAPPED' has to be the first member.
    SOCKET client;           // Use this to pass preemptive socket.
    SOCKET listenSocket;     // Use this to pass the listenSocket.
    DWORD dwBytes;
    SOCKET_OPERATION operation;       // Enum to assist in knowing what socket operation ...
} WSAOVERLAPPEDPLUS, *LPWSAOVERLAPPEDPLUS;

...
typedef enum SOCKET_OPERATION { 
                            UNINITIALIZED_ENUM, // To protect against memory leaks and uninitialized buffers.
                            OP_ACCEPTEX, 
                            OP_RECEIVE, 
                            OP_SEND 
                          };
...

//
// So the previously mentioned for() loop will become;
//

for( /* loop for 1000 times */ ) {
    SOCKET preemptiveSocket = WSASocket(, , , , , WSA_FLAG_OVERLAPPED);
    LPWSAOVERLAPPEDPLUS pOl = new WSAOVERLAPPEDPLUS();

    // Initialize our "extended" overlapped structure
    memset(pOl, 0, sizeof(WSAOVERLAPPEDPLUS));
    pOl->operation = OP_ACCEPTEX;
    pOl->client = preemptiveSocket;
    pOl->listenSocket = listenSocket;

    int buflen = (sizeof(SOCKADDR_IN) + 16) * 2;
    char* pBuf = new char[buflen];
    memset(pBuf, 0, buflen);

    m_lpfnAcceptEx(listenSocket,
                    preemptiveSocket,
                    pBuf,
                    0,  // Passed 0 to avoid reading data on accept which in turn
                        // avoids DDoS attack i.e., connection attempt without data will 
                        // cause AcceptEx to wait indefinitely.
                    sizeof(SOCKADDR_IN) + 16,
                    sizeof(SOCKADDR_IN) + 16,
                    &pOl->dwBytes,
                    &pOl->ProviderOverlapped
                    );
}

... 并且在 GetQueuedCompletionStatus() returns;

时的工作线程中
while (TRUE)
{
    bOk = ::GetQueuedCompletionStatus(hCompPort, &bytes_transferred, &completion_key, &pOverlapped, INFINITE);

    if (bOk) {
        // Process a successfully completed I/O request

        if (completion_key == 0) {
            // Safe way to extract the customized structure from pointer 
            // is to use 'CONTAINING_RECORD'. Read more on 'CONTAINING_RECORD'.
            WSAOVERLAPPEDPLUS *pOl = CONTAINING_RECORD(pOverlapped, WSAOVERLAPPEDPLUS, ProviderOverlapped);

            if (pOl->operation == OP_ACCEPTEX) {
                // Before doing any WSASend/WSARecv, inherit the 
                // listen socket properties by calling 'setsockopt()'
                // as explained earlier.
                // The listenSocket and the preemptive socket are available 
                // in the 'pOl->listenSocket' & 'pOl->client', respectively.

            }
            delete pOl;
        }
    }
    else {
           // Handle error ...
    }

我希望这能让您了解如何将 AcceptEx() 与 IOCP 结合使用。