Преобразование оператора SQL в Spark

Question

Как мне преобразовать этот оператор SQL в Spark SQL?

 replace_old_engagements_sql = """ UPDATE """ + my_table_name + """
                            SET Engagement = CASE Engagement
                                                WHEN '800000026680' THEN '800000032764'
                                                WHEN '807000000041' THEN '808000000000'
                                                WHEN '870000012569' THEN '807000000412'
                                                WHEN '807000000279' THEN '808000000223'
                                                WHEN '807000000282' THEN '808000000223'
                                                WHEN '870000000403' THEN '808000000223'
                                            END
                            WHERE LinkedAccountId in ('123456789101','109876543212') AND Engagement IN ('800000026680', '807000000041', '870000012569', '807000000279', '807000000282', '870000000403'); """

Answer 1

Вы можете использовать операторы case с левым соединением для обновления той же таблицы.

Например: у меня ниже таблица кустов с некоторыми записями теста:

desc test;
OK
account_id              string
country                 string
date_id                 string


789     Korea   20190728
567     Japan   20180612
345     Mexico  20160612
123     USA     20190715
456     Mexico  20190726

spark.sql(""" 
Insert overwrite table db.test
select
 coalesce(acc_num,account_id)
,coalesce(country_code,country)
,coalesce(loaded_date,date_id)
from 
db.test TAB1
LEFT JOIN
(
select
 account_id as acc_num
,CASE 
   WHEN country = 'Mexico' 
   THEN 'Updated'
   WHEN country = 'Korea'
   THEN 'Updated'
   ELSE country
END as country_code
,date_id  as loaded_date
from db.test 
where account_id in (123,456,345,789)
) TAB2
on TAB2.acc_num = TAB1.account_id """)


>>> df=spark.table("db.test")
>>> df.show()
+----------+-------+--------+
|account_id|country| date_id|
+----------+-------+--------+
|       789|Updated|20190728|
|       567|  Japan|20180612|
|       345|Updated|20160612|
|       123|    USA|20190715|
|       456|Updated|20190726|
+----------+-------+--------+

Answer 2

# I hope this could be answer for your query in dataframe style which is creating new dataframe with the modified values as per the conditions specified

val out_df = input_df.filter($"LinkedAccountId".isin('123456789101','109876543212') and $"Engagement".isin('800000026680', '807000000041', '870000012569', '807000000279', '807000000282', '870000000403')).withColumn("Engagement",when($"Engagement" === '800000026680', '800000032764').when($"Engagement" === '807000000041','808000000000').when($"Engagement" === '870000012569', '807000000412').when($"Engagement" === '807000000279', '808000000223').when($"Engagement" === '807000000282', '808000000223').when($"Engagement" === '870000000403', '808000000223').otherwise('0'))