Ошибка приведения типа при приеме данных из PostgreSQL в Druid - PullRequest
Новая
       19

Ошибка приведения типа при приеме данных из PostgreSQL в Druid

0 голосов
/ 10 июля 2019

Я пытаюсь получить данные из PostgreSQL в Druid, используя Firehose.

Я добавил druid.extensions.loadList=["postgresql-metadata-storage"] в файл conf, но задача не выдает

java.lang.ClassCastException: java.util.LinkedHashMap нельзя преобразовать в java.nio.ByteBuffer

файл спецификации приема пищи 

{
  "type": "index",
  "spec": {
      "dataSchema": {
          "dataSource": "dataset_1007",
          "parser": {
              "type": "string",
              "parseSpec": {
                  "format": "tsv",
                  "columns": [
                      "id",
                      "name",
                      "datetimecol"
                  ],
                  "timestampSpec": {
                      "column": "datetimecol",
                      "format": "auto"
                  },
                  "dimensionsSpec": {
                      "dimensions": [
                          "id",
                          "name",
                          "datetimecol"
                      ]
                  }
              }
          },
          "granularitySpec": {
              "type": "uniform",
              "segmentGranularity": "DAY",
              "queryGranularity": "NONE",
              "rollup": false
          }
      },
      "ioConfig": {
          "type": "index",
          "firehose": {
              "type": "sql",
              "database": {
                  "type": "postgresql",
                  "connectorConfig": {
                      "connectURI": "jdbc:postgresql://ISVDRDBILXP1/testdb",
                      "user": "druid",
                      "password": "druid"
                  }
              },
              "sqls": [
                  "SELECT id,name,datetimecol FROM public.testtable"
              ]
          },
          "appendToExisting": false
      },
      "tuningConfig": {
          "forceExtendableShardSpecs": true,
          "type": "index"
      }
  }
}

Действительно трудно выяснить, какой столбец таблицы создает эту проблему, и я изменил весь тип столбца на varchar (). Пожалуйста, укажите, если я где-то ошибаюсь.

Обновление

StackTrace: 

2019-07-10T09:44:10,476 INFO [firehose_fetch_0] org.apache.druid.data.input.impl.prefetch.Fetcher - Fetching [0]th object[SELECT id,name,datetimecol FROM public.testtable], fetchedBytes[0]
2019-07-10T09:44:10,528 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.CustomExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2019-07-10T09:44:10,530 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.ForbiddenExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2019-07-10T09:44:10,530 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding org.apache.druid.server.initialization.jetty.BadRequestExceptionMapper to GuiceManagedComponentProvider with the scope "Singleton"
2019-07-10T09:44:10,531 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.json.JacksonJsonProvider to GuiceManagedComponentProvider with the scope "Singleton"
2019-07-10T09:44:10,538 INFO [main] com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory - Binding com.fasterxml.jackson.jaxrs.smile.JacksonSmileProvider to GuiceManagedComponentProvider with the scope "Singleton"
2019-07-10T09:44:10,636 INFO [task-runner-0-priority-0] org.apache.druid.data.input.impl.prefetch.CacheManager - Object[SELECT id,name,datetimecol FROM public.testtable] is cached. Current cached bytes is [188]
2019-07-10T09:44:10,648 ERROR [task-runner-0-priority-0] org.apache.druid.indexing.common.task.IndexTask - Encountered exception in DETERMINE_PARTITIONS.
java.lang.ClassCastException: java.util.LinkedHashMap cannot be cast to java.nio.ByteBuffer
    at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:31) ~[druid-processing-0.15.0-incubating.jar:0.15.0-incubating]
    at org.apache.druid.data.input.impl.SqlFirehose.nextRow(SqlFirehose.java:68) ~[druid-core-0.15.0-incubating.jar:0.15.0-incubating]
    at org.apache.druid.indexing.common.task.IndexTask.collectIntervalsAndShardSpecs(IndexTask.java:744) ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
    at org.apache.druid.indexing.common.task.IndexTask.createShardSpecsFromInput(IndexTask.java:671) ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
    at org.apache.druid.indexing.common.task.IndexTask.determineShardSpecs(IndexTask.java:606) ~[druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
    at org.apache.druid.indexing.common.task.IndexTask.run(IndexTask.java:437) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
    at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:419) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
    at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:391) [druid-indexing-service-0.15.0-incubating.jar:0.15.0-incubating]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_212]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_212]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_212]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_212]

1 Ответ

0 голосов
/ 12 июля 2019

Если кто ищет ответ.При извлечении данных из SQL мы должны использовать map парсер. Это обновленная спецификация, которую я использую. 

  "parser": {
    "type" : "map",
    "parseSpec": {
      "format": "timeAndDims",
      "dimensionsSpec": {
        "dimensions": [
          "dim1",
          "dim2",
          "dim3"
        ]
      },
      "timestampSpec": {
        "format": "auto",
        "column": "ts"
      }
    }
  }
...