Я немного изменил ваш код и получаю результат
Создан фрейм данных с одной записью с той же схемой, что и у вас
val df = Seq(("select * from test", "type1", "device1", "10.0", "20.0", "1234", "23.4567", "10.98")).toDF.selectExpr("_1 as QUERY", "_2 as TYPE", "_3 as DEVICE", "_4 as PURCHASE_UNITS_SUM", "_5 as CLICK_SUM", "_6 as IMPRESSION_COUNT", "_7 as CLICK_THROUGH_RATE", "_8 as PURCHASE_RATE")
Ниже приведена схема и пример строка
root
|-- QUERY: string (nullable = true)
|-- TYPE: string (nullable = true)
|-- DEVICE: string (nullable = true)
|-- PURCHASE_UNITS_SUM: string (nullable = true)
|-- CLICK_SUM: string (nullable = true)
|-- IMPRESSION_COUNT: string (nullable = true)
|-- CLICK_THROUGH_RATE: string (nullable = true)
|-- PURCHASE_RATE: string (nullable = true)
+------------------+-----+-------+------------------+---------+----------------+------------------+-------------+
| QUERY| TYPE| DEVICE|PURCHASE_UNITS_SUM|CLICK_SUM|IMPRESSION_COUNT|CLICK_THROUGH_RATE|PURCHASE_RATE|
+------------------+-----+-------+------------------+---------+----------------+------------------+-------------+
|select * from test|type1|device1| 10.0| 20.0| 1234| 23.4567| 10.98|
+------------------+-----+-------+------------------+---------+----------------+------------------+-------------+
val result = df.withColumn("CLICK_THROUGH_RATE_MAP", map(col("DEVICE"), col("CLICK_THROUGH_RATE"))).
withColumn("PURCHASE_RATE_MAP", map(col("DEVICE"), col("PURCHASE_RATE"))).
withColumn("PURCHASE_SUM_MAP", map(col("DEVICE"), col("PURCHASE_UNITS_SUM"))).
withColumn("CLICK_SUM_MAP", map(col("DEVICE"), col("CLICK_SUM"))).
withColumn("IMPRESSION_SUM_MAP", map(col("DEVICE"), col("IMPRESSION_COUNT"))).
groupBy("QUERY", "TYPE").
agg(collect_list("CLICK_THROUGH_RATE_MAP"), collect_list("PURCHASE_RATE_MAP"), collect_list("PURCHASE_SUM_MAP"), collect_list("CLICK_SUM_MAP"), collect_list("IMPRESSION_SUM_MAP")).
as[(String, String, Seq[scala.collection.immutable.Map[String, Double]], Seq[scala.collection.immutable.Map[String, Double]], Seq[scala.collection.immutable.Map[String, Double]], Seq[scala.collection.immutable.Map[String, Double]], Seq[scala.collection.immutable.Map[String, Double]])]
result.show
+------------------+-----+------------------------------------+-------------------------------+------------------------------+---------------------------+--------------------------------+
| QUERY| TYPE|collect_list(CLICK_THROUGH_RATE_MAP)|collect_list(PURCHASE_RATE_MAP)|collect_list(PURCHASE_SUM_MAP)|collect_list(CLICK_SUM_MAP)|collect_list(IMPRESSION_SUM_MAP)|
+------------------+-----+------------------------------------+-------------------------------+------------------------------+---------------------------+--------------------------------+
|select * from test|type1| [Map(device1 -> 2...| [Map(device1 -> 1...| [Map(device1 -> 1...| [Map(device1 -> 2...| [Map(device1 -> 1...|
+------------------+-----+------------------------------------+-------------------------------+------------------------------+---------------------------+--------------------------------+
Я изменил функцию карты следующим образом
val finalresultdf = result.map { f => (f._1, f._2, f._3.reduce(_ ++ _), f._4.reduce(_ ++ _), f._5.reduce(_ ++ _), f._6.reduce(_ ++ _), f._7.reduce(_ ++ _)) }.
toDF("QUERY", "TYPE", "CLICK_THROUGH_RATE", "PURCHASE_RATE", "PURCHASE_UNITS", "CLICKS", "IMPRESSIONS")
finalresultdf.show
+------------------+-----+--------------------+--------------------+--------------------+--------------------+--------------------+
| QUERY| TYPE| CLICK_THROUGH_RATE| PURCHASE_RATE| PURCHASE_UNITS| CLICKS| IMPRESSIONS|
+------------------+-----+--------------------+--------------------+--------------------+--------------------+--------------------+
|select * from test|type1|Map(device1 -> 23...|Map(device1 -> 10...|Map(device1 -> 10.0)|Map(device1 -> 20.0)|Map(device1 -> 12...|
+------------------+-----+--------------------+--------------------+--------------------+--------------------+--------------------+
finalresultdf.count
scala> finalresultdf.count
res34: Long = 1
Надеюсь, это поможет !!!