Spark Dataframe Изменение содержимого столбца

Question

У меня есть кадр данных, как показано ниже df.show():

+--------+---------+---------+---------+---------+
|  Col11 |  Col22  | Expend1 | Expend2 | Expend3 |
+--------+---------+---------+---------+---------+
| Value1 | value1  |     123 |    2264 |      56 |
| Value1 | value2  |     124 |    2255 |      23 |
+--------+---------+---------+---------+---------+

Могу ли я преобразовать вышеприведенный фрейм данных в приведенный ниже, используя какой-нибудь SQL?

+--------+---------+-------------+---------------+------------+
| Col11  |  Col22  | Expend1     | Expend2       | Expend3    |
+--------+---------+-------------+---------------+------------+
| Value1 | value1  | Expend1:123 | Expend2: 2264 | Expend3:56 |
| Value1 | value2  | Expend1:124 | Expend2: 2255 | Expend3:23 |
+--------+---------+-------------+---------------+------------+

Answer 1

 val df = Seq(("Value1", "value1",    "123",   "2264",     "56"), ("Value1", "value2",    "124",   "2255",     "23") ).toDF("Col11", "Col22", "Expend1", "Expend2", "Expend3")
 val cols = df.columns.filter(!_.startsWith("Col"))   // It will only fetch other than col% prefix columns

 val getCombineData = udf { (colName:String, colvalue:String) => colName + ":"+ colvalue}
 var in = df 
 for (e <- cols) {
    in = in.withColumn(e, getCombineData(lit(e), col(e)) )
 } 
 in.show

 // results
 +------+------+-----------+------------+----------+
 | Col11| Col22|    Expend1|     Expend2|   Expend3|
 +------+------+-----------+------------+----------+
 |Value1|value1|Expend1:123|Expend2:2264|Expend3:56|
 |Value1|value2|Expend1:124|Expend2:2255|Expend3:23|
 +------+------+-----------+------------+----------+

Answer 2

Вы можете использовать идею foldLeft здесь

import spark.implicits._
import org.apache.spark.sql.functions._

val df = spark.sparkContext.parallelize(Seq(
  ("Value1", "value1", "123", "2264", "56"),
  ("Value1", "value2", "124", "2255", "23")
)).toDF("Col11", "Col22", "Expend1", "Expend2", "Expend3")

//Lists your columns for operation
val cols = List("Expend1", "Expend2", "Expend3")

val newDF = cols.foldLeft(df){(acc, name)  =>
  acc.withColumn(name, concat(lit(name + ":"), col(name)))
}

newDF.show()

Выход:

+------+------+-----------+------------+----------+
| Col11| Col22|    Expend1|     Expend2|   Expend3|
+------+------+-----------+------------+----------+
|Value1|value1|Expend1:123|Expend2:2264|Expend3:56|
|Value1|value2|Expend1:124|Expend2:2255|Expend3:23|
+------+------+-----------+------------+----------+

Answer 3

вы можете сделать это с помощью простого оператора выбора SQL, если вы хотите использовать также udf

Ex -> выберите Col11, Col22, 'Expend1:' + cast (Expend1 как varchar (10)) как Expend1, .... из таблицы