使用 Netty/Kortlin/Ktor 通过 TLS 的 gRPC 会引发错误并阻止 gNMI 工作

gRPC over TLS using Netty/Kortlin/Ktor raises error and prevents gNMI to work

目前正在使用 gRPC/gNMI + Kotlin 和 Ktor HTTP 开发一个模块。 卡在安全依赖库 tc-native netty-tcnative-boringssl-static 2.0.19-Final

Ktor 在 1.0.0-beta3

  // ktor
    implementation "io.ktor:ktor-server-core:${ktor_version}"
    implementation "io.ktor:ktor-server-netty:${ktor_version}"
    implementation "io.ktor:ktor-html-builder:${ktor_version}"
    implementation "io.ktor:ktor-gson:${ktor_version}"
    implementation "io.ktor:ktor-metrics:${ktor_version}"
    implementation "io.ktor:ktor-locations:${ktor_version}"
    implementation "ch.qos.logback:logback-classic:${logback_version}"

    // tpl
    implementation "io.ktor:ktor-freemarker:${ktor_version}"
    implementation "org.freemarker:freemarker:${freemarker_version}"

    // Security
    //implementation "io.netty:netty-tcnative:${netty_tcnative_version}"
    implementation "io.netty:netty-tcnative-boringssl-static:${netty_tcnative_version}".

使用 gRPC 请求时

fun Route.routeFilesystem() {
get("/test") {
    call.respondHtml {
        head {
            title { +"gRPC/gNMI" }
        }
        body {
            p {
                +"[ Capabilities Test ] "
                TLSConnection("xxx.xxx.xxx.xxx",8448).capabilities()
            }
        }
    }
}

它给出了以下错误(堆栈跟踪):

[DEBUG] 2018-11-09 16:50:36.789 [nettyCallPool-4-4] DEBUG i.n.u.internal.NativeLibraryLoader - netty_tcnative_linux_x86_64 cannot be loaded from java.libary.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no netty_tcnative_linux_x86_64 in java.library.path
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
        at java.lang.Runtime.loadLibrary0(Runtime.java:870)
        at java.lang.System.loadLibrary(System.java:1122)
        at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
        at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
        at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
        at io.netty.util.internal.NativeLibraryLoader.loadFirstAvailable(NativeLibraryLoader.java:85)
        at io.netty.handler.ssl.OpenSsl.loadTcNative(OpenSsl.java:440)
        at io.netty.handler.ssl.OpenSsl.<clinit>(OpenSsl.java:97)
        at io.grpc.netty.GrpcSslContexts.defaultSslProvider(GrpcSslContexts.java:244)
        at io.grpc.netty.GrpcSslContexts.configure(GrpcSslContexts.java:171)
        at io.grpc.netty.GrpcSslContexts.forClient(GrpcSslContexts.java:120)
        at sdn.client.cli.console.TLSConnection.<init>(TLSConnection.kt:38)
        at sdn.client.ui.ApplicationKt$routeFilesystem.invoke(Application.kt:125)
        at sdn.client.ui.ApplicationKt$routeFilesystem.invoke(Application.kt)
        at kotlinx.html.ApiKt.visit(api.kt:80)
        at kotlinx.html.Gen_tag_groupsKt.p(gen-tag-groups.kt:125)
        at kotlinx.html.Gen_tag_groupsKt.p$default(gen-tag-groups.kt:125)
        at .sdn.client.ui.ApplicationKt$routeFilesystem.invoke(Application.kt:123)
        at .sdn.client.ui.ApplicationKt$routeFilesystem.invoke(Application.kt)
        at kotlinx.html.ApiKt.visit(api.kt:80)
        at kotlinx.html.Gen_tags_hKt.body(gen-tags-h.kt:164)
        at kotlinx.html.Gen_tags_hKt.body$default(gen-tags-h.kt:164)
        at .sdn.client.ui.ApplicationKt$routeFilesystem.invoke(Application.kt:122)
        at sdn.client.ui.ApplicationKt$routeFilesystem.invoke(Application.kt)
        at kotlinx.html.ApiKt.visit(api.kt:80)
        at kotlinx.html.ApiKt.visitAndFinalize(api.kt:93)
        at kotlinx.html.Gen_consumer_tagsKt.html(gen-consumer-tags.kt:317)
        at kotlinx.html.Gen_consumer_tagsKt.html$default(gen-consumer-tags.kt:317)
        at io.ktor.html.HtmlContent.writeTo(RespondHtml.kt:32)
        at io.ktor.server.engine.BaseApplicationResponse$respondWriteChannelContent.invokeSuspend(BaseApplicationResponse.kt:155)
        at io.ktor.server.engine.BaseApplicationResponse$respondWriteChannelContent.invoke(BaseApplicationResponse.kt)
        at io.ktor.util.cio.ReadersKt.use(Readers.kt:33)
        at io.ktor.server.engine.BaseApplicationResponse.respondWriteChannelContent$suspendImpl(BaseApplicationResponse.kt:149)
        at io.ktor.server.engine.BaseApplicationResponse.respondWriteChannelContent(BaseApplicationResponse.kt)
        at io.ktor.server.engine.BaseApplicationResponse.respondOutgoingContent$suspendImpl(BaseApplicationResponse.kt:119)
        at io.ktor.server.engine.BaseApplicationResponse.respondOutgoingContent(BaseApplicationResponse.kt)
        at io.ktor.server.netty.NettyApplicationResponse.respondOutgoingContent$suspendImpl(NettyApplicationResponse.kt:37)
        at io.ktor.server.netty.NettyApplicationResponse.respondOutgoingContent(NettyApplicationResponse.kt)
        at io.ktor.server.engine.BaseApplicationResponse$$special$$inlined$apply$lambda.invokeSuspend(BaseApplicationResponse.kt:36)
        at io.ktor.server.engine.BaseApplicationResponse$$special$$inlined$apply$lambda.invoke(BaseApplicationResponse.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.PipelineContext.proceedWith(PipelineContext.kt:39)
        at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations.invokeSuspend(DefaultTransform.kt:26)
        at io.ktor.server.engine.DefaultTransformKt$installDefaultTransformations.invoke(DefaultTransform.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:24)
        at io.ktor.html.RespondHtmlKt.respondHtml(RespondHtml.kt:37)
        at io.ktor.html.RespondHtmlKt.respondHtml$default(RespondHtml.kt:15)
        at sdn.client.ui.ApplicationKt$routeFilesystem.invokeSuspend(Application.kt:116)
        at .sdn.client.ui.ApplicationKt$routeFilesystem.invoke(Application.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:24)
        at io.ktor.routing.Routing.executeResult(Routing.kt:110)
        at io.ktor.routing.Routing.interceptor(Routing.kt:29)
        at io.ktor.routing.Routing$Feature$install.invokeSuspend(Routing.kt:75)
        at io.ktor.routing.Routing$Feature$install.invoke(Routing.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.features.ContentNegotiation$Feature$install.invokeSuspend(ContentNegotiation.kt:60)
        at io.ktor.features.ContentNegotiation$Feature$install.invoke(ContentNegotiation.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.features.StatusPages$intercept.invokeSuspend(StatusPages.kt:88)
        at io.ktor.features.StatusPages$intercept.invoke(StatusPages.kt)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:82)
        at kotlinx.coroutines.CoroutineScopeKt.coroutineScope(CoroutineScope.kt:162)
        at io.ktor.features.StatusPages.intercept(StatusPages.kt:87)
        at io.ktor.features.StatusPages$Feature$install.invokeSuspend(StatusPages.kt:121)
        at io.ktor.features.StatusPages$Feature$install.invoke(StatusPages.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.features.CallLogging$Feature$install.invokeSuspend(CallLogging.kt:124)
        at io.ktor.features.CallLogging$Feature$install.invoke(CallLogging.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:24)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline.invokeSuspend(DefaultEnginePipeline.kt:80)
        at io.ktor.server.engine.DefaultEnginePipelineKt$defaultEnginePipeline.invoke(DefaultEnginePipeline.kt)
        at io.ktor.util.pipeline.PipelineContext.proceed(PipelineContext.kt:53)
        at io.ktor.util.pipeline.Pipeline.execute(Pipeline.kt:24)
        at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest.invokeSuspend(NettyApplicationCallHandler.kt:31)
        at io.ktor.server.netty.NettyApplicationCallHandler$handleRequest.invoke(NettyApplicationCallHandler.kt)
        at kotlinx.coroutines.intrinsics.UndispatchedKt.startCoroutineUndispatched(Undispatched.kt:54)
        at kotlinx.coroutines.CoroutineStart.invoke(CoroutineStart.kt:111)
        at kotlinx.coroutines.AbstractCoroutine.start(AbstractCoroutine.kt:160)
        at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch(Builders.common.kt:54)
        at kotlinx.coroutines.BuildersKt.launch(Unknown Source)
        at io.ktor.server.netty.NettyApplicationCallHandler.handleRequest(NettyApplicationCallHandler.kt:22)
        at io.ktor.server.netty.NettyApplicationCallHandler.channelRead(NettyApplicationCallHandler.kt:16)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)
        at io.netty.channel.AbstractChannelHandlerContext.access0(AbstractChannelHandlerContext.java:38)
        at io.netty.channel.AbstractChannelHandlerContext.run(AbstractChannelHandlerContext.java:353)
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:464)
        at io.netty.util.concurrent.SingleThreadEventExecutor.run(SingleThreadEventExecutor.java:884)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
        Suppressed: java.lang.UnsatisfiedLinkError: no netty_tcnative_linux_x86_64 in java.library.path
                at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
                at java.lang.Runtime.loadLibrary0(Runtime.java:870)
                at java.lang.System.loadLibrary(System.java:1122)
                at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
                at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                at java.lang.reflect.Method.invoke(Method.java:498)
                at io.netty.util.internal.NativeLibraryLoader.run(NativeLibraryLoader.java:263)
                at java.security.AccessController.doPrivileged(Native Method)
                at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
                at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
                ... 90 common frames omitted

请问有什么办法可以解决这个问题吗? 谢谢

https://github.com/netty/netty-tcnative/issues/331

这是 DEBUG 级别的打印,这表明它可能不是真正的错误,只是 netty 尝试过但加载失败。加载 boringssl .so 稍后可能会成功。