Spark-NLP 预训练管道是否仅适用于 linux 系统？
Do Spark-NLP pretrained pipelines only work on linux systems?
我正在尝试设置一个简单的代码，我在其中传递数据帧并使用 johnSnowLabs Spark-NLP 库提供的预训练解释管道对其进行测试。
我正在使用 anaconda 的 jupyter 笔记本，并使用 apache toree 设置了 spark scala kernet。每次我 运行 它应该加载预训练管道的步骤时，它都会抛出一个张量流错误。有什么办法可以在 windows 本地 运行 这个吗？
I was trying this in a maven project earlier and the same error had happened. Another colleague tried it on a linux system and it worked. Below is the code I have tried and the error that it gave.
import org.apache.spark.ml.PipelineModel
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
import org.apache.spark.sql.SparkSession
val spark: SparkSession = SparkSession
.builder()
.appName("test")
.master("local[*]")
.config("spark.driver.memory", "4G")
.config("spark.kryoserializer.buffer.max", "200M")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.getOrCreate()
val testData = spark.createDataFrame(Seq(
(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
(2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States"))).toDF("id", "text")
val pipeline = PretrainedPipeline("explain_document_dl", lang = "en") //this is where it gives error
val annotation = pipeline.transform(testData)
annotation.show()
annotation.select("entities.result").show(false)
出现以下错误：
Name: java.lang.UnsupportedOperationException Message: Spark NLP tried
to load a Tensorflow Graph using Contrib module, but failed to load it
on this system. If you are on Windows, this operation is not
supported. Please try a noncontrib model. If not the case, please
report this issue. Original error message:
Op type not registered 'BlockLSTM' in binary running on
'MyMachine'. Make sure the Op and Kernel are registered in the
binary running in this process. Note that if you are loading a saved
graph which used ops from tf.contrib, accessing (e.g.)
tf.contrib.resampler should be done before importing the graph, as
contrib ops are lazily registered when the module is first accessed.
StackTrace: Op type not registered 'BlockLSTM' in binary running on
'MyMachine'. Make sure the Op and Kernel are registered in the
binary running in this process. Note that if you are loading a saved
graph which used ops from tf.contrib, accessing (e.g.)
tf.contrib.resampler should be done before importing the graph, as
contrib ops are lazily registered when the module is first accessed.
at
com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.readGraph(TensorflowWrapper.scala:163)
at
com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.read(TensorflowWrapper.scala:202)
at
com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel$class.readTensorflowModel(TensorflowSerializeModel.scala:73)
at
com.johnsnowlabs.nlp.annotators.ner.dl.NerDLModel$.readTensorflowModel(NerDLModel.scala:134)
at
com.johnsnowlabs.nlp.annotators.ner.dl.ReadsNERGraph$class.readNerGraph(NerDLModel.scala:112)
at
com.johnsnowlabs.nlp.annotators.ner.dl.NerDLModel$.readNerGraph(NerDLModel.scala:134)
at
com.johnsnowlabs.nlp.annotators.ner.dl.ReadsNERGraph$$anonfun.apply(NerDLModel.scala:116)
at
com.johnsnowlabs.nlp.annotators.ner.dl.ReadsNERGraph$$anonfun.apply(NerDLModel.scala:116)
at
com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead.apply(ParamsAndFeaturesReadable.scala:31)
at
com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead.apply(ParamsAndFeaturesReadable.scala:30)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$class.com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead(ParamsAndFeaturesReadable.scala:30)
at
com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read.apply(ParamsAndFeaturesReadable.scala:41)
at
com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read.apply(ParamsAndFeaturesReadable.scala:41)
at
com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:19)
at
com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:8)
at
org.apache.spark.ml.util.DefaultParamsReader$.loadParamsInstance(ReadWrite.scala:652)
at
org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun.apply(Pipeline.scala:274)
at
org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun.apply(Pipeline.scala:272)
at
scala.collection.TraversableLike$$anonfun$map.apply(TraversableLike.scala:234)
at
scala.collection.TraversableLike$$anonfun$map.apply(TraversableLike.scala:234)
at
scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186)
at
scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at
org.apache.spark.ml.Pipeline$SharedReadWrite$.load(Pipeline.scala:272)
at
org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:348)
at
org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:342)
at
com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:135)
at
com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:129)
at com.johnsnowlabs.nlp.pretrained.PretrainedPipelinenter code
heree.(PretrainedPipeline.scala:14)
我检查过，那个管道里面有一个 NER 模型。该 NER 模型是使用 TensorFlow 训练的，它内部有一些 contrib 代码，仅与基于 Unix 的 OS 兼容，例如 Linux 和 macOS。这里有一个未解决的问题：
https://github.com/tensorflow/tensorflow/issues/26468
为此，他们发布了一些名为 Windows 兼容的 管道 noncontrib。您可以将管道名称更改为以下名称：
val pipeline = PretrainedPipeline("explain_document_dl_noncontrib", lang = "en")
所有预训练管道的来源：
https://nlp.johnsnowlabs.com/docs/en/pipelines
完全公开：我是 Spark NLP 库的贡献者之一。
更新：自从 Spark NLP 2.4.0 发布以来，所有模型和管道现在都是跨平台的：https://github.com/JohnSnowLabs/spark-nlp-models
如果您使用的是 Spark NLP 2.4.0 版本，这应该适用于 Linux、macOS 和 Windows：
val pipeline = PretrainedPipeline("explain_document_dl", lang = "en")
更新 2022：除了 M1 和 aarch64 架构（目前），所有 5000+ models/pipelines 都与 Windows 兼容（8 、10 和 11)、Linux（Ubuntu、Debian、CentOS 等）和 macOS 操作系统。
Spark NLP 模型中心：https://nlp.johnsnowlabs.com/models
我正在尝试设置一个简单的代码，我在其中传递数据帧并使用 johnSnowLabs Spark-NLP 库提供的预训练解释管道对其进行测试。 我正在使用 anaconda 的 jupyter 笔记本，并使用 apache toree 设置了 spark scala kernet。每次我 运行 它应该加载预训练管道的步骤时，它都会抛出一个张量流错误。有什么办法可以在 windows 本地 运行 这个吗？
I was trying this in a maven project earlier and the same error had happened. Another colleague tried it on a linux system and it worked. Below is the code I have tried and the error that it gave.
import org.apache.spark.ml.PipelineModel
import com.johnsnowlabs.nlp.pretrained.PretrainedPipeline
import com.johnsnowlabs.nlp.SparkNLP
import org.apache.spark.sql.SparkSession
val spark: SparkSession = SparkSession
.builder()
.appName("test")
.master("local[*]")
.config("spark.driver.memory", "4G")
.config("spark.kryoserializer.buffer.max", "200M")
.config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
.getOrCreate()
val testData = spark.createDataFrame(Seq(
(1, "Google has announced the release of a beta version of the popular TensorFlow machine learning library"),
(2, "Donald John Trump (born June 14, 1946) is the 45th and current president of the United States"))).toDF("id", "text")
val pipeline = PretrainedPipeline("explain_document_dl", lang = "en") //this is where it gives error
val annotation = pipeline.transform(testData)
annotation.show()
annotation.select("entities.result").show(false)
出现以下错误：
Name: java.lang.UnsupportedOperationException Message: Spark NLP tried to load a Tensorflow Graph using Contrib module, but failed to load it on this system. If you are on Windows, this operation is not supported. Please try a noncontrib model. If not the case, please report this issue. Original error message:
Op type not registered 'BlockLSTM' in binary running on 'MyMachine'. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.)
tf.contrib.resamplershould be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. StackTrace: Op type not registered 'BlockLSTM' in binary running on 'MyMachine'. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.)tf.contrib.resamplershould be done before importing the graph, as contrib ops are lazily registered when the module is first accessed.
at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.readGraph(TensorflowWrapper.scala:163) at com.johnsnowlabs.ml.tensorflow.TensorflowWrapper$.read(TensorflowWrapper.scala:202) at com.johnsnowlabs.ml.tensorflow.ReadTensorflowModel$class.readTensorflowModel(TensorflowSerializeModel.scala:73) at com.johnsnowlabs.nlp.annotators.ner.dl.NerDLModel$.readTensorflowModel(NerDLModel.scala:134) at com.johnsnowlabs.nlp.annotators.ner.dl.ReadsNERGraph$class.readNerGraph(NerDLModel.scala:112) at com.johnsnowlabs.nlp.annotators.ner.dl.NerDLModel$.readNerGraph(NerDLModel.scala:134) at com.johnsnowlabs.nlp.annotators.ner.dl.ReadsNERGraph$$anonfun.apply(NerDLModel.scala:116) at com.johnsnowlabs.nlp.annotators.ner.dl.ReadsNERGraph$$anonfun.apply(NerDLModel.scala:116) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead.apply(ParamsAndFeaturesReadable.scala:31) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead.apply(ParamsAndFeaturesReadable.scala:30) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$class.com$johnsnowlabs$nlp$ParamsAndFeaturesReadable$$onRead(ParamsAndFeaturesReadable.scala:30) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read.apply(ParamsAndFeaturesReadable.scala:41) at com.johnsnowlabs.nlp.ParamsAndFeaturesReadable$$anonfun$read.apply(ParamsAndFeaturesReadable.scala:41) at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:19) at com.johnsnowlabs.nlp.FeaturesReader.load(ParamsAndFeaturesReadable.scala:8) at org.apache.spark.ml.util.DefaultParamsReader$.loadParamsInstance(ReadWrite.scala:652) at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun.apply(Pipeline.scala:274) at org.apache.spark.ml.Pipeline$SharedReadWrite$$anonfun.apply(Pipeline.scala:272) at scala.collection.TraversableLike$$anonfun$map.apply(TraversableLike.scala:234) at scala.collection.TraversableLike$$anonfun$map.apply(TraversableLike.scala:234) at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186)
at org.apache.spark.ml.Pipeline$SharedReadWrite$.load(Pipeline.scala:272) at org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:348) at org.apache.spark.ml.PipelineModel$PipelineModelReader.load(Pipeline.scala:342) at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:135) at com.johnsnowlabs.nlp.pretrained.ResourceDownloader$.downloadPipeline(ResourceDownloader.scala:129) at com.johnsnowlabs.nlp.pretrained.PretrainedPipelinenter code heree.(PretrainedPipeline.scala:14)
我检查过，那个管道里面有一个 NER 模型。该 NER 模型是使用 TensorFlow 训练的，它内部有一些 contrib 代码，仅与基于 Unix 的 OS 兼容，例如 Linux 和 macOS。这里有一个未解决的问题：
https://github.com/tensorflow/tensorflow/issues/26468
为此，他们发布了一些名为 Windows 兼容的 管道 noncontrib。您可以将管道名称更改为以下名称：
val pipeline = PretrainedPipeline("explain_document_dl_noncontrib", lang = "en")
所有预训练管道的来源： https://nlp.johnsnowlabs.com/docs/en/pipelines
完全公开：我是 Spark NLP 库的贡献者之一。
更新：自从 Spark NLP 2.4.0 发布以来，所有模型和管道现在都是跨平台的：https://github.com/JohnSnowLabs/spark-nlp-models
如果您使用的是 Spark NLP 2.4.0 版本，这应该适用于 Linux、macOS 和 Windows：
val pipeline = PretrainedPipeline("explain_document_dl", lang = "en")
更新 2022：除了 M1 和 aarch64 架构（目前），所有 5000+ models/pipelines 都与 Windows 兼容（8 、10 和 11)、Linux（Ubuntu、Debian、CentOS 等）和 macOS 操作系统。 Spark NLP 模型中心：https://nlp.johnsnowlabs.com/models