Редактировать : РАЗРЕШЕНО Я думаю, что проблема заключается в многомерных массивах, сгенерированных из вывода Elmo. Я усреднил все векторы и затем использовал окончательный средний вектор для всех слов в предложении в качестве выходных данных, и теперь он работает для преобразования во фрейм данных. Теперь я должен сделать это быстрее, проверим использование потоков.
Попытка использовать предварительно обученную модель ElmoForManyLangs из следующего github для генерации вложений Elmo для предложений во фрейме данных pyspark. Однако я не могу преобразовать полученный объект в фрейм данных.
https://github.com/HIT-SCIR/ELMoForManyLangs
import sys
from pyspark.sql.functions import split
import pandas as pd
import numpy as np
from pyspark.sql.functions import trim
sys.path.append('/tmp/python-elmo/elmoManyLangs/elmoManyLangsGit/ELMoForManyLangs-master')
from elmoformanylangs import Embedder
e = Embedder('/mnt/tmp/python-elmo/elmoManyLangs/english/')
new_list = []
input = spark.read.parquet("/path/to/input/file")
words = input.withColumn("wordlist", split(trim(input["description"]), " ")).dropna().select("product_name","wordlist").limit(1)
wordsPd=words.toPandas()
for t in wordsPd.itertuples():
new_list.append(np.average(np.array([np.average(x,axis=0) for x in e.sents2elmo(t[2])]), axis=0).tolist())
wordsPd = wordsPd.assign(embeddings=new_list)
myDf = spark.createDataFrame(wordsPd)
myDf.registerTempTable("myDf")
wordsPd
0 my_product_name ... 0 [[0.1606223, 0.09298285, -0.3494971, 0.2. .. [1 строка x 3 столбца]
wordsPd.dtypes
product_name object
description object
embeddings object
dtype: object
Вот ошибка для создания кадра данных.
Traceback (most recent call last):
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1068, in _infer_type
return _infer_schema(obj)
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1094, in _infer_schema
raise TypeError("Can not infer schema for type: %s" % type(row))
TypeError: Can not infer schema for type: <class 'object'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1068, in _infer_type
return _infer_schema(obj)
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1096, in _infer_schema
fields = [StructField(k, _infer_type(v), True) for k, v in items]
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1096, in <listcomp>
fields = [StructField(k, _infer_type(v), True) for k, v in items]
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1070, in _infer_type
raise TypeError("not supported type: %s" % type(obj))
TypeError: not supported type: <class 'object'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1068, in _infer_type
return _infer_schema(obj)
.........
.........
raise TypeError("not supported type: %s" % type(obj))
TypeError: not supported type: <class 'pandas.core.indexes.range.RangeIndex'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-7355529425587840217.py", line 360, in <module>
exec(code, _zcUserQueryNameSpace)
...........
...........
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1070, in _infer_type
raise TypeError("not supported type: %s" % type(obj))
TypeError: not supported type: <class 'pandas.core.series.Series'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-7355529425587840217.py", line 367, in <module>
raise Exception(traceback.format_exc())
.........
.........
raise TypeError("Can not infer schema for type: %s" % type(row))
TypeError: Can not infer schema for type: <class 'object'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1068, in _infer_type
return _infer_schema(obj)
.........
.........
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1070, in _infer_type
raise TypeError("not supported type: %s" % type(obj))
TypeError: not supported type: <class 'object'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1068, in _infer_type
return _infer_schema(obj)
........
........
........
........
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1070, in _infer_type
raise TypeError("not supported type: %s" % type(obj))
TypeError: not supported type: <class 'pandas.core.indexes.range.RangeIndex'>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/tmp/zeppelin_pyspark-7355529425587840217.py", line 360, in <module>
exec(code, _zcUserQueryNameSpace)
File "<stdin>", line 17, in <module>
File "/usr/lib/spark/python/pyspark/sql/session.py", line 691, in createDataFrame
rdd, schema = self._createFromLocal(map(prepare, data), schema)
........
........
........
........
File "/usr/lib/spark/python/pyspark/sql/types.py", line 1070, in _infer_type
raise TypeError("not supported type: %s" % type(obj))
TypeError: not supported type: <class 'pandas.core.series.Series'>