У меня есть запись в формате csv, и я импортировал ее как фрейм данных:
---------------------------
name | age | entranceDate |
---------------------------
Tom | 12 | 2019-10-01 |
---------------------------
Mary | 15 | 2019-10-01 |
---------------------------
И когда я использую:
String[] partitions =
new String[] {
"name",
"entranceDate"
};
df.write()
.partitionBy(partitions)
.mode(SaveMode.Append)
.parquet(parquetPath);
Он записывает мой паркет в файл (.parquet).Но странная вещь, когда я снова пытаюсь прочитать с паркета:
public static StructType createSchema() {
final StructType schema = DataTypes.createStructType(Arrays.asList(
DataTypes.createStructField("name", DataTypes.StringType, false),
DataTypes.createStructField("age", DataTypes.StringType, false),
DataTypes.createStructField("entranceDate", DataTypes.StringType, false)
));
return schema;
}
sqlContext.read()
.schema(createSchema())
.parquet(pathToParquet);
.show()
Поле name становится нечитаемым:
| name | age | entranceDate|
+--------------------+----+
|?F...|Tom| 2019-10-01 |
|?F...|Mary| 2019-10-01 |
+--------------------+
Как это возможно?Но я попробовал, если я не поставлю строку .partitionBy(partitions), я могу читать без проблем.
Может кто-нибудь объяснить, в чем причина?Я искал некоторое время, но не нашел причину.
РЕДАКТИРОВАТЬ : Я попытался получить поле "имя" (row.getString(0)), и я получил значениекак показано ниже, но я не могу прочитать это:
?F??m???9??A?Aorg/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter??:??A?Aorg.apache.spark.sql.catalyst.expressions.codegen.UnsafeRowWriter??!:??A?Aorg/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter??7:??A?Aorg/apache/spark/sql/catalyst/expressions/codegen/UnsafeRowWriter?-??9????Q:??A?Forg/apache/spark/sql/catalyst/expressions/BaseGenericInternalRow$class??h:??,??A?Forg.apache.spark.sql.catalyst.expressions.BaseGenericInternalRow$class?????:??A?Forg/apache/spark/sql/catalyst/expressions/BaseGenericInternalRow$class]??6x]???:???:???]??:??????x?:??????b?x?:?????c?x?:?????r?x?:?????c?x?:?????1c?x?:???????x?:?????.??x?:?????Nc?x?:?????]c?x?:????????x?:???????x?:????????x?:???????x?:????????x?:???????xy?x????:??]??X;??T???????:???:??????:???5??x?:???5?.???:???x????:??K0?i?x?i?x??6x6x??6x6x???:??A?Eorg/apache/spark/sql/catalyst/trees/TreeNode$$anonfun$transformDown$2??
;???;?A?Eorg.apache.spark.sql.catalyst.trees.TreeNode$$anonfun$transformDown$2????#;??A?Eorg/apache/spark/sql/catalyst/trees/TreeNode$$anonfun$transformDown$2???j?v9??:???:??:???:??7;??9;???<?;;??>;?????H;???"?@?x?i?xux?]?E;???"?@?x?i?xux????:??;??????:???5??x[;???5?.??[;???x???c;??K0?i?x?i?x??6x6x??6x6x???j?v9?h;???:?h;??[;??s;??u;???<?w;??z;??????;???"?egx?i?xux?]??;???"?egx?i?xux???h;???;??????:???5??b?x?;???5?.???;???b?x????;??K0?i?x?i?x??6x6x??6x6x???j?v9??;???:??;??;??;??;???<??;??;??????;???"?o_x?i?xux?]??;???"?o_x?i?xux????;??<??????:???5?c?x?;???5?.???;??c?x????;??K0?i?x?i?x??6x6x??6x6x???j?v9??;???:??;???;???;???;???<??;???;??????;???"??lx?i?xux?]??;???"??lx?i?xux????;??H<??????:???5?r?x<???5?.??<??r?x???<??K0?i?x?i?x??6x6x??6x6x???j?v9?<???:?<??<??'<??)<???<?+<??.<?????8<???"?;_x?i?xux?]?5<???"?;_x?i?xux???<??<??????:???5?c?xK<???5?.??K<??c?x???S<??K0?i?x?i?x??6x6x??6x6x???j?v9?X<???:?X<??K<??c<??e<???<?g<??j<?????t<???"?H_x?i?xux?]?q<???"?H_x?i?xux???X<???<??????:???5?1c?x?<???5?.???<??1c?x????<??K0?i?x?i?x??6x6x??6x6x???j?v9??<???:??<??<??<??<???<??<??<??????<???"?|_x?i?x?/x?]??<???"?|_x?i?x?/x????<???<??????:???5???x?<???5?.???<???x????<??K0?i?x?i?x??6x6x??6x6x???j?v9??<???:??<???<???<???<???<??<???<??????<???"??_x?i?x?/x?]??<???"??_x?i?x?/x????<??8=??????:???5?.??x?<???5?.???<??.??x???=??K0?i?x?i?x??6x6x??6x6x???j?v9?=???:?=???<??=??=???<?=??=?????(=???"?T_x?i?xux?]?%=???"?T_x?i?xux???=??t=??????:???5?Nc?x;=???5?.??;=??Nc?x???C=??K0?i?x?i?x??6x6x??6x6x???j?v9?H=???:?H=??;=??S=??U=???<?W=??Z=?????d=???"?{lx?i?xux?]?a=???"?{lx?i?xux???H=??=??????:???5?]c?xw=???5?.??w=??]c?x???=??K0?i?x