У меня есть простой массив json, и я могу читать его в spark-dataframe.
Можете ли вы помочь обернуть эти столбцы в пользовательский корневой тег.
точнее, прямо напротив опции разнесения, ограничивающей целые строки данных в пользовательском целевом корневом столбце.
Initial Json Data:
[{"tpeKeyId":"301461865","acctImplMgrId":null,"acctMgrId":null,"agreCancDt":null,"agreEffDt":null,"pltfrmNm":"EMPLOYEE NAVIGATOR","premPyRmtInd":null,"recCrtTs":"2016-11-08 13:01:44.290418","recCrtUsrId":"testedname","recUpdtTs":"2018-10-16 12:16:21.579446","recUpdtUsrId":"testname","spclInstrFormCd":null,"sysCd":null,"tpeNm":"EMPLOYEE NAVIGATOR","univPrdcrId":"9393939393"},{"tpeKeyId":"901972280","acctImplMgrId":null,"acctMgrId":null,"agreCancDt":null,"agreEffDt":null,"pltfrmNm":"datalion","premPyRmtInd":null,"recCrtTs":"2018-12-10 01:36:14.925833","recCrtUsrId":"exactlydata","recUpdtTs":"2018-12-10 01:36:14.925833","recUpdtUsrId":"datalion ","spclInstrFormCd":null,"sysCd":null,"tpeNm":"lialion","univPrdcrId":"89899898989"}]
First Dataframe:
+-------------+---------+----------+---------+------------------+------------+--------------------------+-----------+--------------------------+----------------+---------------+-----+---------+------------------+-----------+
|acctImplMgrId|acctMgrId|agreCancDt|agreEffDt|pltfrmNm |premPyRmtInd|recCrtTs |recCrtUsrId|recUpdtTs |recUpdtUsrId |spclInstrFormCd|sysCd|tpeKeyId |tpeNm |univPrdcrId|
+-------------+---------+----------+---------+------------------+------------+--------------------------+-----------+--------------------------+----------------+---------------+-----+---------+------------------+-----------+
|null |null |null |null |EMPLOYEE NAVIGATOR|null |2016-11-08 13:01:44.290418|testedname |2018-10-16 12:16:21.579446|testname |null |null |301461865|EMPLOYEE NAVIGATOR|9393939393 |
|null |null |null |null |datalion |null |2018-12-10 01:36:14.925833|exactlydata|2018-12-10 01:36:14.925833|datalion |null |null |901972280|lialion |89899898989|
+-------------+---------+----------+---------+------------------+------------+--------------------------+-----------+--------------------------+----------------+---------------+-----+---------+------------------+-----------+
После объединения корневого тега вручную:
val addingRootTag= "{ \"roottag\" :" + fileContents + "}"
val rootTagDf = spark.read.json(Seq(addingRootTag).toDS())
rootTagDf.show(false)
Second Dataframe:
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|roottag |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[[,,,, EMPLOYEE NAVIGATOR,, 2016-11-08 13:01:44.290418, testedname, 2018-10-16 12:16:21.579446, testname,,, 301461865, EMPLOYEE NAVIGATOR, 9393939393], [,,,, datalion,, 2018-12-10 01:36:14.925833, exactlydata, 2018-12-10 01:36:14.925833, datalion ,,, 901972280, lialion, 89899898989]]|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Вопрос в том, есть ли у нас такой метод в API, поддерживаемом spark-framework, чтобы избежать ручной конкатенации roottag и получения first-dataframe , обернутого для построения, отображаемого как second dataframe ? EXACTLY OPPOSITE TO EXPLODE OPTION