Spark - я установил sparkcontext на пряжу, но она не работает

Question

Я установил sparkcontext в режим пряжи, но он не работает.

Я пытаюсь собрать искровой фрейм данных по шоу, но отображается сообщение об ошибке.

Можете ли вы помочь мне?

Сообщение об ошибке:

Py4JJavaError: An error occurred while calling o117.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 2.0 failed 4 times, most recent failure: Lost task 0.3 in stage 2.0 (TID 13, xxxxxxxxxxxxxxxxxxxxxxxxxxxx, executor 1): org.apache.spark.SparkException: Python worker exited unexpectedly (crashed)

Сценарий ниже.

import pyspark
from pyspark.sql import SQLContext, SparkSession
from pyspark.sql.functions import pandas_udf, PandasUDFType
from pyspark.sql.types import *
from pyspark.ml.feature import Word2Vec
from pyspark.ml import Pipeline
import pandas as pd
from pyspark.ml.feature import Tokenizer
from pyspark import SparkConf
from pyspark import SparkContext
sc = SparkContext(appName="W2V", master="yarn")
spark = SparkSession(sc)

df_sentencas_tratadas = spark.read.option("header", "true").csv('hdfs://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/arquivo.csv).limit(1)
schema = StructType([StructField('indice',StringType(), True),StructField('sentencas_tratadas',StringType(), True)])

@pandas_udf(schema, PandasUDFType.GROUPED_MAP)
def string2list(df):    
    sent = df['sentencas_tratadas'][0]  
    sent = sent.replace("'", ""). replace("[", ""). replace(",", ""). replace("]", "")
    return pd.DataFrame({'_c0':df['_c0'][0], 'sentencas_tratadas':sent}, index=[0])


df = df_sentencas_tratadas.groupby('_c0').apply(string2list)

tokenizer = Tokenizer(inputCol="sentencas_tratadas", outputCol="tokens")
w2v = Word2Vec(vectorSize=100, minCount=0, inputCol="tokens", outputCol="vetor")

df = tokenizer.transform(df)

df.show(1)`