Не удается разрешить 'sport_details' для 1 входного столбца в Spark

Question

Я пытаюсь определить схему для своего json в Spark. У меня эта ошибка:

org. apache .spark. sql .AnalysisException: не удается разрешить 'sport_details' заданные столбцы ввода: [user_agent, modified_at, datamatrix_content, player_id, created_by, device_advertisers_id, device_type, betslips, created_by_app, created_by_user, status, device_id, uid, created_at, created_by_partner] ;;

Вот мой код:

      val mySchema = StructType(Array(
    StructField("uid", StringType, true),
    StructField("modified_at", TimestampType, true),
    StructField("created_at", TimestampType, true),
    StructField("datamatrix_content", StringType, true),
    StructField("created_by", StringType, true),
    StructField("created_by_app", StringType, true),
    StructField("created_by_partner", StringType, true),
    StructField("created_by_user", StringType, true),
    StructField("status", StringType, true),
    StructField("player_id", StringType, true),
    StructField("device_type", StringType, true),
    StructField("device_id", StringType, true),
    StructField("user_agent", StringType, true),

    StructField("betslips", StructType(Array(
      StructField("game_name", StringType, true),
      StructField("type_id", IntegerType, true),
      StructField("expires_at", TimestampType, true),
      StructField("enabled", BooleanType, true),


      StructField("sport_details", StructType(Array(
        StructField("grid_id", IntegerType, true),
        StructField("double_count", IntegerType, true),
        StructField("triple_count", IntegerType, true)
      ))))))))

И вот мой json:

[
  {
    "uid": "*",
    "modified_at": "*",
    "created_at": "*",
    "datamatrix_content": "string",
    "created_by": "string",
    "created_by_app": "*",
    "created_by_partner": "string",
    "created_by_user": "string",
    "status": "pending",
    "player_id": "string",
    "device_type": "mobile/ios",
    "device_id": "string",
    "device_advertisers_id": "string",
    "user_agent": "string",

    "betslips": [
      {
        "game_name": "*",
        "type_id": 0,
        "expires_at": "*",
        "enabled": true,
 
        "sport_details": [
          {
            "grid_id": 122,
            "double_count": 2,
            "triple_count": 3
          }
        ]
      }
    ]
  }
]

Answer 1

купон и данные пользователя - это ArrayType в вашем json, поэтому вы должны добавить их в схему, как показано ниже

val mySchema = StructType(Array(
  StructField("uid", StringType, true),
  StructField("modified_at", TimestampType, true),
  StructField("created_at", TimestampType, true),
  StructField("datamatrix_content", StringType, true),
  StructField("created_by", StringType, true),
  StructField("created_by_app", StringType, true),
  StructField("created_by_partner", StringType, true),
  StructField("created_by_user", StringType, true),
  StructField("status", StringType, true),
  StructField("player_id", StringType, true),
  StructField("device_type", StringType, true),
  StructField("device_id", StringType, true),
  StructField("user_agent", StringType, true),
  StructField("betslips", ArrayType(
    StructType(Array(
      StructField("game_name", StringType, true),
      StructField("type_id", IntegerType, true),
      StructField("expires_at", TimestampType, true),
      StructField("enabled", BooleanType, true),
      StructField("sport_details", ArrayType(
        StructType(Array(
          StructField("grid_id", IntegerType, true),
          StructField("double_count", IntegerType, true),
          StructField("triple_count", IntegerType, true)
        ))
      ))
    ))
  ))
))

Answer 2

Я полагаю, вы указали неверную схему. купон имеет тип ArrayType, а затем тип элемента - StructType. Кроме того, json, похоже, начинается с анонимной коллекции.

Правильная схема должна быть:

val mySchema = StructType(
            Array(
                StructField("uid", StringType, true),
                StructField("modified_at", TimestampType, true),
                StructField("created_at", TimestampType, true),
                StructField("datamatrix_content", StringType, true),
                StructField("created_by", StringType, true),
                StructField("created_by_app", StringType, true),
                StructField("created_by_partner", StringType, true),
                StructField("created_by_user", StringType, true),
                StructField("status", StringType, true),
                StructField("player_id", StringType, true),
                StructField("device_type", StringType, true),
                StructField("device_id", StringType, true),
                StructField("user_agent", StringType, true),
            
                StructField("betslips", ArrayType(
                    StructType(
                        Array(
                            StructField("game_name", StringType, true),
                            StructField("type_id", IntegerType, true),
                            StructField("expires_at", TimestampType, true),
                            StructField("enabled", BooleanType, true),
                        
                            ArrayType("sport_details", 
                                StructType(Array(
                                    StructField("grid_id", IntegerType, true),
                                    StructField("double_count", IntegerType, true),
                                    StructField("triple_count", IntegerType, true)
                                ))
                            )
                        )
                    )
                ))
            )
        )