groupBy (). count.show (), выдающий ошибку java.lang.IllegalStateException в pyspark

Question

Я пытаюсь показать результаты () с помощью функции groupBy () на моем СДР.Это дает следующую ошибку:

Py4JJavaError: An error occurred while calling o14287.showString.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 3 in stage 2669.0 failed 1 times, most recent failure: Lost task 3.0 in stage 2669.0 (TID 3896, localhost, executor driver): java.lang.IllegalStateException: Input row doesn't have expected number of values required by the schema. 26 fields are required while 1 values are provided.
    at org.apache.spark.sql.execution.python.EvaluatePython$$anonfun$makeFromJava$15$$anonfun$apply$15.applyOrElse(EvaluatePython.scala:184)
    at org.apache.spark.sql.execution.python.EvaluatePython$.org$apache$spark$sql$execution$python$EvaluatePython$$nullSafeConvert(EvaluatePython.scala:208)
    at org.apache.spark.sql.execution.python.EvaluatePython$$anonfun$makeFromJava$15.apply(EvaluatePython.scala:180)

My Pyspark Script :

import pyspark
from pyspark.sql import SparkSession

spark=SparkSession.builder.getOrCreate()

s3RDD=spark.sparkContext.textFile("file:///Users/mydir/Documents/Projects/Pyspark/MiscScripts/logfile.gz")

firstLine = s3RDD.first()

sparkContext.parallelize convert string into RDD
parallelize = spark.sparkContext.parallelize([firstLine])

s3RDD=s3RDD.subtract(parallelize)
s3RDD=s3RDD.map(lambda x: x.split('\t'))

urlsDf=s3RDD.toDF()

#import pyspark.sql.functions as f

urlsDf.groupBy("_8").count().show() 

Answer 1

Это мой текстовый файл:

Версия: 1.0

Поля: дата, время, x-край-местоположение, sc-байты, c-ip, cs-метод cs (Host) cs-uri-stem sc-status cs (Referer) cs (User-Agent) cs-uri-query cs (Cookie) x-edge-result-type x-edge-request-id x-host-header cs-протокол cs-байты time-принято x-forwarded-for ssl-протокол ssl-шифр x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields

2018-04-12 23:55:43MAA50-C1 89352 39.44.14.521 GET mycdn.com / mydir / new-my-dir-url-опасных 200 - Mozilla / 5,0% 2520 (Windows% 2520NT% 25206,1;% 2520WOW64;% 2520rv: 40,0)% 2520Gecko / 20100101%2520Firefox / 40,0 = идентификатор FOxPJf3rutG1qhi - Мисс CZeom7P2yw7bYn5veotj8gS2GpDTWkxZdUDiJHFwBFPSusCXKC4j == mydomain.com HTTP 370 3,169 10.130.24.151,% 2010.140.65.140 - - Мисс HTTP / 1.1 - - 2018-04-12 23:55:51 MAA50-C1-81103 39.44.14.521GET mycdn.com / mydir / mydir-new-test1 200 - Mozilla / 5.0% 2520 (Windows% 2520NT% 25206.1;% 2520WOW64;% 2520rv: 40.0)% 2520Gecko / 20100101% 2520Firefox / 40.0 id = QOP645KHxGQcgXW -Miss 1wKt5erjuDVQNa7X-D - vKQeli3X1ZvE5g32D0H7vgLnq_aiVuNqDA == mydomain.com http 349 1.245 10.130.24.151,% 2010.140.65.140 - - Мисс HTTP / 1.1 - - 2018-04-12 23:55:59 MAA50c-0 0.com / MYDIR / MYDIR-новой test1 000 - Mozilla / 5.0% 2520 (Windows% 2520NT% 25206,1;% 2520WOW64;% 2520rv: 40,0)% 2520Gecko / 20100101% 2520Firefox / 40,0 = идентификатор OCjtSXeh7QwqLtE - Ошибка 8c9OnlJYo_2jI6mBCMFNbtxv7NSV00NjjANS2r7ODqhAlkV3Ew-4AA ==mydomain.com http 371 19.992 10.130.24.151,% 2010.140.65.140 - - Ошибка HTTP / 1.1 - - 2018-04-12 23:55:45 BOM52 64704 103.18.142.29 GET mycdn.com / mydir / mydir-new-test1 200- Mozilla / 5.0% 2520 (Macintosh;% 2520Intel% 2520Mac% 2520OS% 2520X% 252010_9_5)% 2520AppleWebKit / 537,36% 2520 (KHTML,% 2520like% 2520Gecko)% 2520Chrome / 42.0.2311.90% 2520Safari / 537,36 - - RefreshHit UcaCxr82_Wgm-VZETVv0pxhCvoMAjO46JATyF8mBAZ0VPnGmFKGn-A == mydomain.com http 312 0.022 - - - RefreshHit HTTP / 1.1 - - 2018-04-12 23:56:38 SIN2 71625 13.228.207.150 GET mycdn.com / 200 - Mozilla / 5.0% 2520 (Windows;%2520U;% 2520Windows% 2520NT% 25206,0;% 2520en-США;% 2520rv: 1.9.1.6)% 2520Gecko / 20091201% 2520Firefox / 3.5.6% 2520GTB5 - - Мисс 5fGTvqY4-2DWBMPEvOOtaskdX-yPiwEu8RlR4fKfDRwLRetKYIlA == mydomain.com HTTP 233 2,959 - - - Мисс HTTP /1.1 - - 2018-04-12 23:55:41 MAA50-C1 67805 39.44.14.521 GET mycdn.com / mydir / mydir-new-test1 200 - Mozilla / 5,0% 2520 (Windows% 2520NT% 25206,1;% 2520WOW64;%2520rv: 40.0)% 2520Gecko / 20100101% 2520Firefox / 40.0 id = EyTPvato7qq0qiW - Мисс ZPtOvMKzHCvdS-HbAMsSTU5FfYzSmP8xnxM7KAHseJaZFMd6Cykmain040 - 201.1.240.130.18.18.18.0.101.1: 55: 52 MAA50-C1 62402 39.44.14.521 GET mycdn.com / mydir / mydir-new-test1 200 - Mozilla / 5,0% 2520 (Windows% 2520NT% 25206,1;% 2520WOW64;% 2520rv: 40,0)% 2520Gecko / 20100101%2520Firefox / 40.0 id = uGcBwdJhQC2V5sx - мисс 4DDdtWO63B8OBw5JQ29IDv5mdcTJVVLQ0R5PvBbv6YPQSNitxwSuaw == mydomain.com http 356 2.675 10.130.24405 / HTTP 100% - 100% - 100% - 1%