Результат строки Json из Elasticsearch отсутствует в поле запроса Grafana - PullRequest
Новая
       27

Результат строки Json из Elasticsearch отсутствует в поле запроса Grafana

0 голосов
/ 19 сентября 2019

Я пытаюсь собрать данные конечной точки jmx и сохранить в Elasticsearch с помощью плагина logstash HTTP_poller.

На самом деле я могу получить данные из конечной точки jmx с помощью этой команды curl 

  curl  http://{namenodeurl}:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo

, который возвратил 

    {
  "beans" : [ {
    "name" : "Hadoop:service=NameNode,name=NameNodeInfo",
    "modelerType" : "org.apache.hadoop.hdfs.server.namenode.FSNamesystem",
    "LiveNodes" : "{\"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":362336252306,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":12526554222,\"capacity\":30697676811776,\"numBlocks\":23319,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":362336252306,\"remaining\":30321440162507,\"blockScheduled\":10,\"blockPoolUsed\":362336252306,\"blockPoolUsedPercent\":1.1803378,\"volfails\":0,\"lastBlockReport\":276},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":333276152686,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":44941272562,\"capacity\":30697676811776,\"numBlocks\":22068,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":333276152686,\"remaining\":30318573786938,\"blockScheduled\":3,\"blockPoolUsed\":333276152686,\"blockPoolUsedPercent\":1.0856723,\"volfails\":0,\"lastBlockReport\":55},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":357572743864,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":10682192614,\"capacity\":30697676811776,\"numBlocks\":22839,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":357572743864,\"remaining\":30328135188399,\"blockScheduled\":6,\"blockPoolUsed\":357572743864,\"blockPoolUsedPercent\":1.1648201,\"volfails\":0,\"lastBlockReport\":318},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":446027838708,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":8842906256,\"capacity\":30697676811776,\"numBlocks\":24188,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":446027838708,\"remaining\":30241817778097,\"blockScheduled\":7,\"blockPoolUsed\":446027838708,\"blockPoolUsedPercent\":1.4529693,\"volfails\":0,\"lastBlockReport\":154},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":470151363788,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":3429561855,\"capacity\":30697676811776,\"numBlocks\":23687,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":470151363788,\"remaining\":30223312381102,\"blockScheduled\":5,\"blockPoolUsed\":470151363788,\"blockPoolUsedPercent\":1.5315535,\"volfails\":0,\"lastBlockReport\":122},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":467854437056,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":3788540824,\"capacity\":30697676811776,\"numBlocks\":42952,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":467854437056,\"remaining\":30224970092959,\"blockScheduled\":4,\"blockPoolUsed\":467854437056,\"blockPoolUsedPercent\":1.5240711,\"volfails\":0,\"lastBlockReport\":242},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":487515877803,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":4637728137,\"capacity\":30697676811776,\"numBlocks\":23948,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":487515877803,\"remaining\":30204294432356,\"blockScheduled\":9,\"blockPoolUsed\":487515877803,\"blockPoolUsedPercent\":1.5881197,\"volfails\":0,\"lastBlockReport\":302},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":445760514597,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":8830301147,\"capacity\":30697676811776,\"numBlocks\":23442,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":445760514597,\"remaining\":30241955503429,\"blockScheduled\":6,\"blockPoolUsed\":445760514597,\"blockPoolUsedPercent\":1.4520986,\"volfails\":0,\"lastBlockReport\":132},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":413280001149,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":6956526088,\"capacity\":30697676811776,\"numBlocks\":31147,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":413280001149,\"remaining\":30276667590983,\"blockScheduled\":6,\"blockPoolUsed\":413280001149,\"blockPoolUsedPercent\":1.3462908,\"volfails\":0,\"lastBlockReport\":275},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":489923596801,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":9775171728,\"capacity\":30697676811776,\"numBlocks\":24346,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":489923596801,\"remaining\":30196722998834,\"blockScheduled\":13,\"blockPoolUsed\":489923596801,\"blockPoolUsedPercent\":1.5959631,\"volfails\":0,\"lastBlockReport\":74},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":353332585866,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":26225755269,\"capacity\":30697676811776,\"numBlocks\":38301,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":353332585866,\"remaining\":30316940221716,\"blockScheduled\":8,\"blockPoolUsed\":353332585866,\"blockPoolUsedPercent\":1.1510075,\"volfails\":0,\"lastBlockReport\":280}\":3995494748,\"capacity\":30697676811776,\"numBlocks\":23410,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":463225862340,\"remaining\":30229497125844,\"blockScheduled\":6,\"blockPoolUsed\":463225862340,\"blockPoolUsedPercent\":1.5089931,\"volfails\":0,\"lastBlockReport\":265},\"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":466469075307,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":17933366421,\"capacity\":30697676811776,\"numBlocks\":23597,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":466469075307,\"remaining\":30212106669507,\"blockScheduled\":6,\"blockPoolUsed\":466469075307,\"blockPoolUsedPercent\":1.5195582,\"volfails\":0,\"lastBlockReport\":50},
    \"datanodeurl:1019\":{\"infoAddr\":\"x.x.x.x:1022\",\"infoSecureAddr\":\"x.x.x.x:0\",\"xferaddr\":\"x.x.x.x:1019\",\"lastContact\":0,\"usedSpace\":349656174247,\"adminState\":\"In Service\",\"nonDfsUsedSpace\":2078169551,\"capacity\":30697676811776,\"numBlocks\":22250,\"version\":\"2.7.3.2.6.5.23-1\",\"used\":349656174247,\"remaining\":30344393318603,\"blockScheduled\":13,\"blockPoolUsed\":349656174247,\"blockPoolUsedPercent\":1.1390314,\"volfails\":0,\"lastBlockReport\":82}}",
    "DeadNodes" : "{}",
    "DecomNodes" : "{}",
    "BlockPoolId" : "BP-132486844-x.x.x.x1-1553681769814",
    "NameDirStatuses" : "{\"active\":{\"/mnt/hadoop/hadoop/hdfs/namenode\":\"IMAGE_AND_EDITS\"},\"failed\":{}}",
    "NodeUsage" : "{\"nodeUsage\":{\"min\":\"1.09%\",\"median\":\"1.51%\",\"max\":\"1.65%\",\"stdDev\":\"0.19%\"}}",
    "NameJournalStatus" : "[{\"manager\":\"QJM to [x.x.x.x1:8485, x.x.x.x2:8485, x.x.x.x3:8485]\",\"stream\":\"Writing segment beginning at txid 19662635. \\nx.x.x.x1:8485 (Written txid 19664711), x.x.x.x2:8485 (Written txid 19664711), x.x.x.x3:8485 (Written txid 19664711)\",\"disabled\":\"false\",\"required\":\"true\"},{\"manager\":\"FileJournalManager(root=/mnt/hadoop/hadoop/hdfs/namenode)\",\"stream\":\"EditLogFileOutputStream(/mnt/hadoop/hadoop/hdfs/namenode/current/edits_inprogress_0000000000019662635)\",\"disabled\":\"false\",\"required\":\"false\"}]",
    "JournalTransactionInfo" : "{\"MostRecentCheckpointTxId\":\"19635524\",\"LastAppliedOrWrittenTxId\":\"19664711\"}",
    "NNStarted" : "Tue Sep 03 10:27:36 ICT 2019",
    "CompileInfo" : "2018-08-06T13:45Z by jenkins from (detached from 130841f)",
    "CorruptFiles" : "[]",
    "DistinctVersionCount" : 1,
    "DistinctVersions" : [ {
      "key" : "2.7.3.2.6.5.23-1",
      "value" : 16
    } ],
    "SoftwareVersion" : "2.7.3.2.6.5.23-1",
    "RollingUpgradeStatus" : null,
    "Threads" : 1060
  } ]

Результаты выглядят неплохо с Kibana, но, к сожалению, в запросе Grafana нет поля «LiveNodes» (но оно показано в Kibana).Я не уверен, что неправильный формат ввода (поле LiveNodes имеет формат json, а не объект json) вызывает эту проблему, или Grafana не поддерживает этот тип ввода.

Вот мои настройки Logstash 

input {
  http_poller {
    urls => {
      namenode_jvmmetrics => {
        method => get
        user => ""
        password => ""
        url => "http://namenode:50070/jmx?qry=Hadoop:service=NameNode,name=NameNodeInfo"
        headers => {
          Accept => "application/json"
        }
      }
    }
    request_timeout => 60
    schedule => { cron => "* * * * * UTC"}
    codec => "json"
  }
}

filter {
    split {
      field => "beans"
    }
    json {
      source => "beans.LiveNodes"
      target => "LiveNodes"
   }
}

output {
  elasticsearch {
    hosts => ["http://elasticsearchhost01:9200", "http://elasticsearchhost02:9200", "http://elasticsearchhost03:9200"]
  }
}

Я что-то упустил или какие-то неправильные конфигурации в Logstash?Любое предложение было бы хорошо.

Заранее спасибо

...