Сглаживание и чтение значения из столбца данных типа Struct в Spark - PullRequest
Новая
       4

Сглаживание и чтение значения из столбца данных типа Struct в Spark

0 голосов
/ 17 марта 2020

У меня есть набор данных в формате паркета, подобный этому: 

parquetFile.toDF().registerTempTable("tempTable")
val PDataFrame = sqlContext.sql("SELECT * FROM tempTable")
PDataFrame.show()

+--------------------+--------------------+-------------------+-----+--------+-------------------+--------------------+
|                 _id|     VehicleDetailId|             PlanID| Type| SubType|          CreatedOn|                Date|
+--------------------+--------------------+-------------------+-----+--------+-------------------+--------------------+
|[($oid,5cc8e1a72f...|[($numberLong,219...|[($numberLong,164)]|Quote|Response|5/1/2019 5:30:39 AM|[($date,155666883...|
|[($oid,5cc8e1a72f...|[($numberLong,219...|[($numberLong,168)]|Quote|Response|5/1/2019 5:30:39 AM|[($date,155666883...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,102)]|  IDV| Request|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,105)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,112)]|Quote| Request|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,134)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,114)]|Quote| Request|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,115)]|Quote| Request|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,113)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,185)]|Quote| Request|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,108)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,149)]|Quote| Request|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,135)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,167)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,116)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,156)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,125)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,102)]|  IDV|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,144)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,171)]|Quote|Response|5/1/2019 5:30:44 AM|[($date,155666884...|
+--------------------+--------------------+-------------------+-----+--------+--------------------+-------------------+--------------------+
only showing top 20 rows

Схема этого набора данных: 

PDataFrame.printSchema()
root
 |-- _id: struct (nullable = true)
 |    |-- $oid: string (nullable = true)
 |-- VehicleDetailId: struct (nullable = true)
 |    |-- $numberLong: string (nullable = true)
 |-- PlanID: struct (nullable = true)
 |    |-- $numberLong: string (nullable = true)
 |-- Type: string (nullable = true)
 |-- SubType: string (nullable = true)
 |-- CreatedOn: string (nullable = true)
 |-- Date: struct (nullable = true)
 |    |-- $date: string (nullable = true)

Я пытаюсь написать код Spark SQL, используя Scala для чтения данных по значению PlanID в предложении where. Вот почему я хочу использовать запрос SQL по Spark SQL. Вот моя ожидаемая структура вывода (примерное представление из 10 строк) 

+-----------------------+--------------------+-------+-----+--------+-------------------+--------+
|                    _id|     VehicleDetailId| PlanID| Type| SubType|          CreatedOn|    Date|
+-----------------------+--------------------+-------+-----+--------+-------------------+--------+
5ae7ae00b07ccf35c020e5ba|10220998|135|Quote|Response|5/1/2018 5:30:00 AM|1525132800096
5ae7ae00b07ccf35c020e5bb|10220998|134|Quote|Response|5/1/2018 5:30:00 AM|1525132800139
5ae7ae00b07ccf35c020e5bc|10220998|104|Quote|Response|5/1/2018 5:30:00 AM|1525132800516
5ae7ae00b07ccf35c020e5bd|10220998|104|Quote|Response|5/1/2018 5:30:00 AM|1525132800519
5ae7ae00b07ccf35c020e5be|10220998|101|Quote|Response|5/1/2018 5:30:00 AM|1525132800539
5ae7ae00b07ccf35c020e5bf|10220998|103|IDV|Request|5/1/2018 5:30:00 AM|1525132800546
5ae7ae00b07ccf35c020e5c0|10220998|105|Quote|Response|5/1/2018 5:30:00 AM|1525132800577
5ae7ae00b07ccf35c020e5c1|10220998|103|IDV|Request|5/1/2018 5:30:00 AM|1525132800581
5ae7ae00b07ccf35c020e5c2|10220998|103|IDV|Response|5/1/2018 5:30:00 AM|1525132800702
5ae7ae00b07ccf35c020e5c3|10220998|128|Quote|Response|5/1/2018 5:30:00 AM|1525132800709

Теперь я попытался с различным подходом получить ожидаемый результат, например: 

PDataFrame.withColumn("first", $"PlanID.$$numberLong").show

ИЛИ 

sqlContext.sql(s""" select _id["$$oid"] as col1, PlanID["$numberLong"] as col2 from tempTable """)

К сожалению, я не могу достичь ожидаемого результата. Любая помощь будет высоко оценена.

Ответы [ 2 ]

1 голос
/ 17 марта 2020

Я использовал функцию обрезки для достижения этой цели. 

parquetFile.withColumn("first", trim($"PlanID.$$numberLong", "($numberLong,')'")).show

Вывод: 

+--------------------+--------------------+-------------------+-----+--------+--------------------+-------------------+--------------------+-----+
|                 _id|     VehicleDetailId|             PlanID| Type| SubType|                 XML|          CreatedOn|                Date|first|
+--------------------+--------------------+-------------------+-----+--------+--------------------+-------------------+--------------------+-----+
|[($oid,5cc8e1a72f...|[($numberLong,219...|[($numberLong,164)]|Quote|Response|<?xml version="1....|5/1/2019 5:30:39 AM|[($date,155666883...|  164|
|[($oid,5cc8e1a72f...|[($numberLong,219...|[($numberLong,168)]|Quote|Response|<?xml version="1....|5/1/2019 5:30:39 AM|[($date,155666883...|  168|
|[($oid,5cc8e1ac2f...|[($numberLong,219...|[($numberLong,102)]|  IDV| Request|<IDV><policy_star...|5/1/2019 5:30:44 AM|[($date,155666884...|  102|
1 голос
/ 17 марта 2020

Из вашей схемы DataFrame 

 |-- PlanID: struct (nullable = true)
 |    |-- $numberLong: string (nullable = true)

Значение $"PlanID.$$numberLong" равно ($numberLong,164), и это строка. Таким образом, вы должны разделить и выбрать то, что вы хотите. 

PDataFrame.withColumn("first", split($"PlanID.$$numberLong", ",")(1)).show
...