У меня есть DF, как показано ниже, мне нужно преобразовать его в подкадры данных без использования createOrReplaceTempView (и выполнять подзапросы в sql), поскольку подкадры данных будут использоваться для выполнения нескольких левых объединений с мастер-таблицей
Я пробовал listvaluesDF.createOrReplaceTempView ("listvaluesDF") && var dfYear = spark.sql ("ключ выбора, значение из listvaluesDF, где internal = 'year'"), но когда я объясняю его создание представления, которое может конфликтоватьпри большом левом присоединении
+--------------------+--------------------+----------------+
| key| value| internal|
+--------------------+--------------------+----------------+
| accessories| D31 - Accessories| wmt0Department|
| annualEvent| Annual Event| wmt0Event|
| apparel| Apparel| wmt0SBU|
| automotive| D10 - Automotive| wmt0Department|
| baking| baking|wmt0DeptCategory|
| baking| Baking| seasonType|
| baking| Baking|wmt0DeptCategory|
| bathandShower|D20 - Bath and Sh...| wmt0Department|
| bedding| D22 - Bedding| wmt0Department|
| betty| Betty|wmt0DeptCategory|
| boxedCards| Boxed Cards|wmt0DeptCategory|
| boyswear| Boyswear|wmt0DeptCategory|
| boyswear| D24 - Boyswear| wmt0Department|
| buildaBasket| Build a Basket|wmt0DeptCategory|
| camerasAndSupplies|D06 - Cameras & S...| wmt0Department|
| cardsAndGifts| Cards & Gifts| seasonType|
| cardsAndGifts| Cards & Gifts|wmt0DeptCategory|
| carving| Carving|wmt0DeptCategory|
| celebration| D67 - Celebration| wmt0Department|
| chineseNewYear| Chinese New Year| seasonType|
| christmas| Christmas| seasonType|
| cookandDine| D14 - Cook and Dine| wmt0Department|
| costumes| Costumes|wmt0DeptCategory|
| crafts| D44 - Crafts| wmt0Department|
| decor| Décor|wmt0DeptCategory|
| decortiveGarland|Decorative Wreath...|wmt0DeptCategory|
| dotcomOnly| Dotcom Only| wmt0DotcomOnly|
| dressUp| Dress Up|wmt0DeptCategory|
| easter| Easter| seasonType|
| electronics| D72 - Electronics| wmt0Department|
| entertainment| Entertainment| wmt0SBU|
| fall| S3 - Fall| seasonType|
| fallGMPad| Fall GM Pad| wmt0Event|
| fallModular| Fall Modular| seasonType|
| familySocks| D27 - Family Socks| wmt0Department|
| familySocks| Family Socks|wmt0DeptCategory|
| fathersDay| Father's Day| seasonType|
| feature| Feature| wmt0Event|
| featureQ1| Feature Q1| seasonType|
| featureQ2| Feature Q2| seasonType|
| featureQ3| Feature Q3| seasonType|
| featureQ4| Feature Q4| seasonType|
| featureSeason1| Feature Season 1| seasonType|
| featureSeason2| Feature Season 2| seasonType|
| featureSeason3| Feature Season 3| seasonType|
| featureSeason4| Feature Season 4| seasonType|
| firstHalf| H1 – First Half| seasonType|