Ошибка записи данных в таблицу куста с помощью pyspark

Question

Я получаю ошибку ниже при выполнении кода ниже в pyspark.

spark.sql("SELECT Trim(world_srvc_id) AS world_srvc_id,emp_eff_strt_dt AS emp_eff_strt_dt,emp_eff_end_dt  AS emp_eff_end_dt,bus_site_id AS bus_site_id,dsgnt_cd AS dsgnt_cd,regexp_replace(Trim(emp_first_nm),'[^0-9a-zA-Z]',' ') AS emp_first_nm,regexp_replace(Trim(emp_mid_nm),'[^0-9a-zA-Z]',' ') AS emp_mid_nm, regexp_replace(Trim(emp_lst_nm),'[^0-9a-zA-Z]',' ') AS emp_lst_nm,Trim(mgr_world_srvc_id) AS mgr_world_srvc_id,Trim(cas_id) AS cas_id,CASE WHEN ((Trim(desktop_id) = '' AND Trim(cas_id) > ' ')) THEN (Upper(Trim(cas_id))) ELSE (Trim(desktop_id)) END AS desktop_id, func_incntv_care_grp_cd AS func_incntv_care_grp_cd,Cast(lang_id_tx AS String) AS lang_id_tx FROM table1")

Ошибка:

20/06/20 05:16:44 WARN TaskSetManager: Lost task 0.0 in stage 4.0 (TID 20, lgpbd152c.gso.aexp.com, executor 1): org.apache.spark.SparkException: Task failed while writing rows.
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:288)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:197)
        at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:196)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
        at org.apache.spark.scheduler.Task.run(Task.scala:109)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ArrayIndexOutOfBoundsException: 7