Не могу использовать авро данные из тем Confluent Kafka в Hbase - PullRequest
Новая
       18

Не могу использовать авро данные из тем Confluent Kafka в Hbase

0 голосов
/ 19 мая 2018

Я использую приведенный ниже пример кода для загрузки некоторых тестовых данных из Confluent Kafka в Hbase, но почему-то я не вижу данных, поступающих в Hbase.Я перепробовал все, но не работал вообще.Кто-нибудь может помочь в этом?

Как дублировать проблему:

  1. Загрузить образец кода:

https://github.com/mravi/kafka-connect-hbase

Скомпилируйте пример кода

mvn -DskipTests = true пакет

Изменены свойства hbase-sink.properties на серверах ConnectDistributed

vim ./etc/kafka-connect-hbase/hbase-sink.properties 

name=kafka-cdc-hbase
connector.class=io.svectors.hbase.sink.HBaseSinkConnector
tasks.max=1
topics=test
zookeeper.quorum=wdsjnl012.test.com:2181
event.parser.class=io.svectors.hbase.parser.AvroEventParser

hbase.test.rowkey.columns=id

hbase.test.rowkey.delimiter=,

hbase.test.family=name

Start ConnectDistributed следующей командой:

. / Bin / confluent start connect

Создать таблицу Hbase:

hbase (main): 007: 0> создать «тест», «имя»

hbase (основной): 007: 0> описать «тест» Таблица ВКЛЮЧЕНА
тест
ОПИСАНИЕ КОЛОННЫХ СЕМЕЙ
{NAME => 'name', BLOOMFILTER => 'ROW', VERSIONS => '1', IN_MEMORY => 'false', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_EN CODING => 'NONE', TTL => 'FOREVER', COMPRESSION => 'NONE', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}

производить данные:

. / Bin / kafka-avro-console-продуцента --broker-list wdsjnl010: 9092 --топический тест --property value.schema = '{"type": "record", "name": "record", "fields": [{"name": "id", "type": "int"}, {"name": "name", "type": "string"}]} '

{"id": 1, "name": "foo"}

Кстати, я дважды проверяю, загружен ли hbase connect или нет из файла журнала, я думаюHBaseподключаемый плагин уже загружен: 

[2018-05-18 19:48:32,714] INFO Loading plugin from: 
/usr/local/confluent/share/java/kafka-connect-hbase 
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:194)
[2018-05-18 19:48:35,447] INFO Registered loader: 
PluginClassLoader{pluginLocation=file:/usr/local/confluent/share/java/kafka- 
connect-hbase/} 
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:217)
[2018-05-18 19:48:35,447] INFO Added plugin 
'io.svectors.hbase.sink.HBaseSinkConnector' 
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:146)
[2018-05-18 19:48:35,447] INFO Loading plugin from: 
/usr/local/confluent/share/java/confluent-common 
(org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:194)
[2018-05-18 19:48:35,516] INFO Registered loaded:
PluginClassLoader{pluginLocation=file:/usr/local/confluent/share/java/confluent-common/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:217)
    [2018-05-18 19:48:35,518] INFO Loading plugin from: /usr/local/confluent/share/java/kafka-connect-s3 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:194)
[2018-05-18 19:48:38,396] INFO Registered loader: PluginClassLoader{pluginLocation=file:/usr/local/confluent/share/java/kafka-connect-s3/} (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:217)
[2018-05-18 19:48:38,396] INFO Added plugin 'io.confluent.connect.s3.S3SinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:146)
[2018-05-18 19:48:40,673] INFO Registered loader: sun.misc.Launcher$AppClassLoader@764c12b6 (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:217)
[2018-05-18 19:48:40,675] INFO Added aliases 'ElasticsearchSinkConnector' and 'ElasticsearchSink' to plugin 'io.confluent.connect.elasticsearch.ElasticsearchSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,675] INFO Added aliases 'HdfsSinkConnector' and 'HdfsSink' to plugin 'io.confluent.connect.hdfs.HdfsSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,676] INFO Added aliases 'JdbcSinkConnector' and 'JdbcSink' to plugin 'io.confluent.connect.jdbc.JdbcSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,676] INFO Added aliases 'JdbcSourceConnector' and 'JdbcSource' to plugin 'io.confluent.connect.jdbc.JdbcSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,676] INFO Added aliases 'S3SinkConnector' and 'S3Sink' to plugin 'io.confluent.connect.s3.S3SinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,676] INFO Added aliases 'HBaseSinkConnector' and 'HBaseSink' to plugin 'io.svectors.hbase.sink.HBaseSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,676] INFO Added aliases 'FileStreamSinkConnector' and 'FileStreamSink' to plugin 'org.apache.kafka.connect.file.FileStreamSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,677] INFO Added aliases 'FileStreamSourceConnector' and 'FileStreamSource' to plugin 'org.apache.kafka.connect.file.FileStreamSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,677] INFO Added aliases 'MockConnector' and 'Mock' to plugin 'org.apache.kafka.connect.tools.MockConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,677] INFO Added aliases 'MockSinkConnector' and 'MockSink' to plugin 'org.apache.kafka.connect.tools.MockSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,677] INFO Added aliases 'MockSourceConnector' and 'MockSource' to plugin 'org.apache.kafka.connect.tools.MockSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,677] INFO Added aliases 'VerifiableSinkConnector' and 'VerifiableSink' to plugin 'org.apache.kafka.connect.tools.VerifiableSinkConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,677] INFO Added aliases 'VerifiableSourceConnector' and 'VerifiableSource' to plugin 'org.apache.kafka.connect.tools.VerifiableSourceConnector' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,678] INFO Added aliases 'AvroConverter' and 'Avro' to plugin 'io.confluent.connect.avro.AvroConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,678] INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,678] INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,678] INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,678] INFO Added aliases 'ByteArrayConverter' and 'ByteArray' to plugin 'org.apache.kafka.connect.converters.ByteArrayConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,678] INFO Added aliases 'JsonConverter' and 'Json' to plugin 'org.apache.kafka.connect.json.JsonConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,678] INFO Added alias 'SimpleHeaderConverter' to plugin 'org.apache.kafka.connect.storage.SimpleHeaderConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:348)
[2018-05-18 19:48:40,678] INFO Added aliases 'StringConverter' and 'String' to plugin 'org.apache.kafka.connect.storage.StringConverter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:351)
[2018-05-18 19:48:40,680] INFO Added alias 'RegexRouter' to plugin 'org.apache.kafka.connect.transforms.RegexRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:348)
[2018-05-18 19:48:40,680] INFO Added alias 'TimestampRouter' to plugin 'org.apache.kafka.connect.transforms.TimestampRouter' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:348)
[2018-05-18 19:48:40,680] INFO Added alias 'ValueToKey' to plugin 'org.apache.kafka.connect.transforms.ValueToKey' (org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader:348)
[2018-05-18 19:48:40,700] INFO DistributedConfig values:
...