无法在 pyspark 中加载模型并使用 grpc 提供服务

Question

我正在尝试使用 grpc 提供火花酒分类模型。

我已经训练好了模型。而且我已经编写了使用 Flask 在 http 服务器上提供服务的代码。

现在，我正在尝试使用 grpc 为其提供服务，以尝试了解性能提升。

但是我收到以下错误 -

服务器端 -

ERROR:root:Exception calling application: classify() missing 1 required positional argument: 'context'
Traceback (most recent call last):
  File "/media/dosi/Data/mlframework/venv/grpc/lib/python3.5/site-packages/grpc/_server.py", line 375, in _call_behavior
    return behavior(argument, context), True
TypeError: classify() missing 1 required positional argument: 'context'

客户端 -

Traceback (most recent call last):
  File "client.py", line 20, in <module>
    run()
  File "client.py", line 16, in run
    proline=23.12))
  File "/media/dosi/Data/mlframework/venv/grpc/lib/python3.5/site-packages/grpc/_channel.py", line 507, in __call__
    return _end_unary_response_blocking(state, call, False, deadline)
  File "/media/dosi/Data/mlframework/venv/grpc/lib/python3.5/site-packages/grpc/_channel.py", line 455, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNKNOWN, Exception calling application: classify() missing 1 required positional argument: 'context')>

我有以下疑问，但我认为这不是错误的原因 -

grpc 安装在 python 虚拟环境中。也许，在执行 spark-submit 时我没有使用那个虚拟环境。我不认为这是问题所在，因为在这种情况下我不应该能够导入 grpc 更不用说启动服务器和运行客户端了。

服务器代码-

import time, sys,  os
from pyspark import SparkContext, SparkConf
from pyspark.sql import SparkSession, SQLContext
import config
import grpc
import wine_pb2
import wine_pb2_grpc
from concurrent import futures

_ONE_DAY_IN_SECONDS = 60 * 60 * 24

class WineClassifier(wine_pb2_grpc.WineClassifierServicer):

  def classify(self, request, context):
    return wine_pb2.WineClass(request.mg)

def init_spark_context():
    # load spark context
    conf = SparkConf().setAppName("wine-app-server")
    # IMPORTANT: pass aditional Python modules to each worker
    sc = SparkContext(conf=conf)
    spark = SparkSession \
    .builder \
    .appName("Python Spark SQL basic example") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()

    return sc, spark


def run_server():
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
    wine_pb2_grpc.add_WineClassifierServicer_to_server(WineClassifier, server)
    server.add_insecure_port('[::]:50051')
    server.start()
    try:
        while True:
            time.sleep(_ONE_DAY_IN_SECONDS)
    except KeyboardInterrupt:
        server.stop(0)

if __name__ == "__main__":
    # Init spark context and load libraries
    sc, spark = init_spark_context()

    # start web server
    run_server()

原始文件-

syntax="proto3";

service WineClassifier {
  rpc classify (WineAttributes) returns (WineClass) {}
}

message WineAttributes {
    double alcohol = 1;
    double malic_acid = 2;
    double ash = 3;
    double alcalinity = 4;
    int32 mg = 5;
    double phenols = 6;
    double flavanoids = 7;
    double flavaniods = 8;
    double nf_phenols = 9;
    double proanthocyanins = 10;
    double color_intensity = 11;
    double hue = 12;
    double diluted = 13;
    double proline = 14;
}

message WineClass {
  int32 class = 1;
}

知道这里出了什么问题吗？

Answer 1

我认为问题在于，在 wine_pb2_grpc.add_WineClassifierServicer_to_server(WineClassifier, server) 中，您传递的是 WineClassifier class 对象，而不是 WineClassifier class 的实例。尝试将行更改为 wine_pb2_grpc.add_WineClassifierServicer_to_server(WineClassifier(), server).

无法在 pyspark 中加载模型并使用 grpc 提供服务

Unable to load a model in pyspark and serve using grpc

python

apache-spark

pyspark

grpc