NGINX 代理不适用于 HTTP/2 和 gRPC

NGINX Proxy not working with HTTP/2 and gRPC

我正在尝试使用 NGINX 作为“API 网关”进入我的 gRPC 服务——所有这些都在 Kubernetes 集群中。 Typescript React App 只是通过 grpc-web 模块调用 Envoy 代理,然后调用 API NGINX 代理。 (我已经测试了堆栈的那一端 - 我 100% 确定特使工作正常)。

注意: 我可能犯了一个错误 NOT 将 TLS 与 Envoy 代理一起使用('client' NGINX) - 所以如果这是我犯的错误请评论

为了与我的 gRPC 端点一起使用,我需要启用 HTTP/2 代理(这是 gRPC 工作所必需的 - 它必须超过 HTTP/2)。因此,按照此处的官方 NGINX 文档:https://www.nginx.com/blog/nginx-1-13-10-grpc/,我的 nginx.conf 文件如下所示:

worker_processes auto;

events {}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent"';

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''        close;
    }

    server {
        listen 1449 ssl http2;

        ssl_protocols TLSv1.2;
        ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
        ssl_prefer_server_ciphers on;

        ssl_certificate ./server.crt;
        ssl_certificate_key ./server.key;

        location /com.example.grpcService {
            grpc_pass grpcs://api-grpc-server:9090;

            proxy_buffer_size          512k;
            proxy_buffers              4 256k;
            proxy_busy_buffers_size    512k;
            grpc_set_header Upgrade $http_upgrade;
            grpc_set_header Connection "Upgrade";
            grpc_set_header Connection keep-alive;
            grpc_set_header Host $host:$server_port;
            grpc_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            grpc_set_header X-Forwarded-Proto $scheme;
        }
    }
}

我还从另一个论坛听说,您必须将 TLS/SSL 与 HTTP/2 一起使用,否则它不会起作用,所以我首先尝试了它 - 它没有起作用。然后我用生成的 SSL 证书尝试了它,看起来我仍然从代理服务中收到 400 错误。日志看起来像:

172.17.0.17 - - [05/Jan/2021:18:16:23 +0000] "PRI * HTTP/2.0" 400 157 "-" "-"

我已将 OpenSSL 用于生成 .crt.key 文件的证书 - 然后我将其用于我的 Spring Boot gRPC 服务器和 NGINX 代理。我的 OpenSSL 版本是 OpenSSL 1.1.1c 28 May 2019.

我在实际的 gRPC 服务器本身上使用这些相同的证书,这看起来像:

@Component
public class GrpcServerRunner implements CommandLineRunner, DisposableBean {

    private final ConfigurableApplicationContext applicationContext;
    private Server server;

    public GrpcServerRunner(@Autowired ConfigurableApplicationContext applicationContext) {
        this.applicationContext = applicationContext;
    }

    @Override
    public void run(String... args) throws Exception {

        File cert = new File("~/etc/ssl/server.crt");
        File key = new File("~/etc/ssl/server.key");

        BindableService service = applicationContext.getBean("grpcService", BindableService.class);
        server = ServerBuilder.forPort(9090).useTransportSecurity(cert, key).addService(service).build();

        runSever();
    }

    private void runSever() {
        Thread thread = new Thread(() -> {
            try {
                server.awaitTermination();
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        });
        thread.setDaemon(false);
        thread.start();
    }

    @Override
    public void destroy() {
        server.shutdown();
    }

}

对于此问题的任何帮助、问题、反馈或解决方案,我将不胜感激 - 在此先致谢。

它实际上与 gRPC 服务器或 Java 项目无关。 这是根 NGINX 配置文件:

worker_processes auto;
events {
  worker_connections 1024;
}

http {
    log_format main '$remote_addr [$time_local] [$time_local] [$cookie_X-AUTH-TOKEN] '
    '"$scheme $host $request" $status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for" '
    '($request_time)'
    '(($sent_http_set_cookie))';

    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''        close;
    }

    # Upstream servers here
    upstream api-server-address {
        server api-server-address:9090;
        keepalive 20;
    }

    # gRPC Client requirements set
    client_max_body_size 0;
    proxy_request_buffering off;

    server {
        listen 1449 http2;
        include ./config/grpc-header-config.conf.conf;

        # gRPC service proxied here
        location /com.yourpackage {
            auth_request_set    $upstream_http_set_cookie;
            auth_request_set    $upstream_http_status;

            grpc_pass grpc://api-service-address;
            include config/grpc-header-config.conf;
        }
        default_type application/grpc;
    }
}

使这项工作成功的关键文件是这个(这是 config/grpc-header-config.conf 中根文件引用的文件):

error_page 400 = @grpc_internal;
error_page 401 = @grpc_unauthenticated;
error_page 403 = @grpc_permission_denied;
error_page 404 = @grpc_unimplemented;
error_page 429 = @grpc_unavailable;
error_page 502 = @grpc_unavailable;
error_page 503 = @grpc_unavailable;
error_page 504 = @grpc_unavailable;
error_page 405 = @grpc_internal;
error_page 408 = @grpc_deadline_exceeded;
error_page 413 = @grpc_resource_exhausted;
error_page 414 = @grpc_resource_exhausted;
error_page 415 = @grpc_internal;
error_page 426 = @grpc_internal;
error_page 495 = @grpc_unauthenticated;
error_page 496 = @grpc_unauthenticated;
error_page 497 = @grpc_internal;
error_page 500 = @grpc_internal;
error_page 501 = @grpc_internal;



location @grpc_deadline_exceeded {
    add_header          'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
    add_header          'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header          'Set-Cookie' $auth_cookie;
    add_header          'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
    add_header          'Access-Control-Allow-Origin' *';
    add_header          'Access-Control-Allow-Credentials' 'true';
    add_header          'grpc-status' 4;
    add_header          'grpc-message' 'deadline exceeded';
    return              204;
}


location @grpc_permission_denied {
    add_header          'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
    add_header          'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header          'Set-Cookie' $auth_cookie;
    add_header          'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
    add_header          'Access-Control-Allow-Origin' '*';
    add_header          'Access-Control-Allow-Credentials' 'true';
    add_header          'grpc-status' 7;
    add_header          'grpc-message' 'permission denied';
    return              204;
}


location @grpc_resource_exhausted {
    add_header          'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
    add_header          'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header          'Set-Cookie' $auth_cookie;
    add_header          'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
    add_header          'Access-Control-Allow-Origin' '*';
    add_header          'Access-Control-Allow-Credentials' 'true';
    add_header          'grpc-status' 8;
    add_header          'grpc-message' 'resource exhausted';
    return              204;
}


location @grpc_unimplemented {
    add_header          'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
    add_header          'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header          'Set-Cookie' $auth_cookie;
    add_header          'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
    add_header          'Access-Control-Allow-Origin' '*';
    add_header          'Access-Control-Allow-Credentials' 'true';
    add_header          'grpc-status' 12;
    add_header          'grpc-message' unimplemented;
    return              204;
}


location @grpc_internal {
    add_header          'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
    add_header          'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header          'Set-Cookie' $auth_cookie;
    add_header          'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
    add_header          'Access-Control-Allow-Origin' '*';
    add_header          'Access-Control-Allow-Credentials' 'true';
    add_header          'grpc-status' 13;
    add_header          'grpc-message' 'internal error';
    return              204;
}


location @grpc_unavailable {
    add_header          'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
    add_header          'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header          'Set-Cookie' $auth_cookie;
    add_header          'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
    add_header          'Access-Control-Allow-Origin' '*';
    add_header          'Access-Control-Allow-Credentials' 'true';
    add_header          'grpc-status' 14;
    add_header          'grpc-message' 'unavailable';
    return              204;
}


location @grpc_unauthenticated {
    add_header          'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type,Content-Transfer-Encoding,Custom-Header-1,X-Accept-Content-Transfer-Encoding,X-Accept-Response-Streaming,X-User-Agent,X-Grpc-Web,Access-Control-Allow-Credentials';
    add_header          'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header          'Set-Cookie' $auth_cookie;
    add_header          'Access-Control-Expose-Headers' 'Content-Transfer-Encoding,Grpc-Message,Grpc-Status';
    add_header          'Access-Control-Allow-Origin' '*';
    add_header          'Access-Control-Allow-Credentials' 'true';
    add_header          'grpc-status' 16;
    add_header          'grpc-message' '401. Unauthorized.';
    return              200;
}

我意识到这看起来很棒 sketchy/hacky,但这是我唯一能做到的。随时改进这个答案!

您基本上将默认协议设置为 gRPC 和 HTTP/2,然后在任何错误页面上您只需重置状态以匹配 gRPC 约定 + 规范,以便您的客户端能够解析二进制文件。如果你使用 SSL,你只需要照常将证书放在每一边,然后将 grpc_pass 更改为 grpcs://api-server-address 而不是我所拥有的。

请随时添加任何建设性反馈或任何问题!干杯,本