Pandas Dataframe имеет два столбца hero_sku (строка) и neighbors_list (список строк)
>>> visual_skus_to_knns_df.head()
hero_sku neighbors_list
0 IVBX6548 [IVBX6548, IVBX6511, IVBX6535, IVBX6391, IVBX6...
1 IVBX6549 [IVBX6549, IVBX6512, IVBX6536, IVBX6448, IVBX6...
2 BLMS1270 [BLMS1270, FRUP1958, SADL1011, BLMK3080, BLMK4...
3 EUNH6179 [EUNH6179, ETUB7716, URBH6598, BLGA1031, FAV19...
4 IVBX6540 [IVBX6540, IVBX6515, IVBX6502, IVBX6552, IVBX5...
Использование parquet-tools для получения информации о схеме / головке из файла parquet:
$ parquet-tools schema /tmp/rmore/pyarrow_example/demo_rupesh.parquet
message schema {
optional binary hero_sku (UTF8);
optional group neighbors_list (LIST) {
repeated group list {
optional binary item (UTF8);
}
}
optional int64 __index_level_0__;
}
$ parquet-tools head /tmp/rmore/pyarrow_example/demo_rupesh.parquet
hero_sku = IVBX6548
neighbors_list:
.list:
..item = IVBX6548
.list:
..item = IVBX6511
.list:
..item = IVBX6535
.list:
..item = IVBX6391
.list:
..item = IVBX6488
.list:
..item = IVBX6460
.list:
..item = IVBX6475
.list:
..item = IVBX6380
.list:
..item = IVBX6402
.list:
..item = IVBX6502
.list:
..item = IVBX6393
.list:
..item = IVBX5206
.list:
..item = IVBX6526
.list:
..item = IVBX6412
.list:
..item = IVBX6389
.list:
..item = IVBX6425
.list:
..item = IVBX6446
.list:
..item = IVBX6540
.list:
..item = IVBX6515
.list:
..item = IVBX6414
.list:
..item = IVBX5035
__index_level_0__ = 0
Создан стол Hive:
CREATE EXTERNAL TABLE `rmore.hd_visual_skus_to_knns_1`(
`hero_sku` string,
`neighbor_list` array<string>)
STORED AS PARQUET
LOCATION
'/tmp/rmore/pyarrow_example';
При выборе из таблицы кустов столбец соседей-списков отображается как NULL:
hive>
> select * from rmore.hd_visual_skus_to_knns limit 2;
OK
IVBX6548 NULL
IVBX6549 NULL
Time taken: 0.149 seconds, Fetched: 2 row(s)
В prestodb я вижу данные в столбце соседей-списках, и Hive, и PrestoDB совместно используют одно и то же метастроение. Мы используем версию Presto CLI 0.198.
Запрос PrestoDB
Вопрос: Почему Hive не может выбрать данные столбца, где может Presto?
Я также попытался разбить столбец neigbors_list на строки, но снова не смог выбрать разнесенный столбец в улье.
Мы используем версию Hive 1.1.0 на Cloudera 5.14.2.
Также заметил, что после нажатия на улей сеанса, вижу кучу сбоев:
hive>
>
> Aug 31, 2018 9:25:27 AM WARNING: parquet.CorruptStatistics: Ignoring statistics because created_by could not be parsed (see PARQUET-251): parquet-cpp version 1.3.2-SNAPSHOT
parquet.VersionParser$VersionParseException: Could not parse created_by: parquet-cpp version 1.3.2-SNAPSHOT using format: (.+) version ((.*) )?\(build ?(.*)\)
at parquet.VersionParser.parse(VersionParser.java:112)
at parquet.CorruptStatistics.shouldIgnoreStatistics(CorruptStatistics.java:66)
at parquet.format.converter.ParquetMetadataConverter.fromParquetStatistics(ParquetMetadataConverter.java:294)
at parquet.format.converter.ParquetMetadataConverter.fromParquetMetadata(ParquetMetadataConverter.java:601)
at parquet.format.converter.ParquetMetadataConverter.readParquetMetadata(ParquetMetadataConverter.java:578)
at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:431)
at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:386)
at parquet.hadoop.ParquetFileReader.readFooter(ParquetFileReader.java:372)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.getSplit(ParquetRecordReaderWrapper.java:252)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:95)
at org.apache.hadoop.hive.ql.io.parquet.read.ParquetRecordReaderWrapper.<init>(ParquetRecordReaderWrapper.java:81)
at org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat.getRecordReader(MapredParquetInputFormat.java:72)
at org.apache.hadoop.hive.ql.exec.FetchOperator$FetchInputFormatSplit.getRecordReader(FetchOperator.java:674)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:324)
at org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
at org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:2069)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:246)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:175)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:389)
at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:781)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:699)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:634)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
Aug 31, 2018 9:25:27 AM WARNING: parquet.hadoop.ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
Aug 31, 2018 9:25:27 AM INFO: parquet.hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 258319 records.
Aug 31, 2018 9:25:27 AM INFO: parquet.hadoop.InternalParquetRecordReader: at row 0. reading next block
Aug 31, 2018 9:25:27 AM INFO: parquet.hadoop.InternalParquetRecordReader: block read in memory in 216 ms. row count = 258319
Aug 31, 2018 9:25:46 AM WARNING: parquet.hadoop.ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
Aug 31, 2018 9:25:46 AM INFO: parquet.hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 258319 records.
Aug 31, 2018 9:25:46 AM INFO: parquet.hadoop.InternalParquetRecordReader: at row 0. reading next block
Aug 31, 2018 9:25:46 AM INFO: parquet.hadoop.InternalParquetRecordReader: block read in memory in 158 ms. row count = 258319
Aug 31, 2018 9:26:31 AM WARNING: parquet.hadoop.ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
Aug 31, 2018 9:26:31 AM INFO: parquet.hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 258319 records.
Aug 31, 2018 9:26:31 AM INFO: parquet.hadoop.InternalParquetRecordReader: at row 0. reading next block
Aug 31, 2018 9:26:31 AM INFO: parquet.hadoop.InternalParquetRecordReader: block read in memory in 87 ms. row count = 258319
Aug 31, 2018 9:26:39 AM WARNING: parquet.hadoop.ParquetRecordReader: Can not initialize counter due to context is not a instance of TaskInputOutputContext, but is org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl
Aug 31, 2018 9:26:39 AM INFO: parquet.hadoop.InternalParquetRecordReader: RecordReader initialized will read a total of 258319 records.
Aug 31, 2018 9:26:39 AM INFO: parquet.hadoop.InternalParquetRecordReader: at row 0. reading next block
Aug 31, 2018 9:26:40 AM INFO: parquet.hadoop.InternalParquetRecordReader: block read in memory in 87 ms. row count = 258319