В настоящее время я проверяю, почему пользовательский интерфейс APM в Кибане не выводит мне информацию, когда таймфрейм установлен более чем на 24 часа.
При проверке конфигурации я заметил, что у нас не установлена конечная точка Kibana для сервера APM.
Проверка журналов APM Я не вижу ошибки, но при переходе к интерфейсу APM я могу найти эту ошибку в журналах Kibana:
kibana |{"type":"response","@timestamp":"2020-01-23T09:15:08Z","tags":["access:apm"],"pid":7,"method":"get","statusCode":400,"req":{"url":"/api/apm/settings/agent-configuration/services","method":"get","headers":{"host":"<our hostname>","connection":"keep-alive","sec-fetch-dest":"empty","kbn-version":"7.3.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.53 Safari/537.36 Edg/80.0.361.33","content-type":"application/json","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"https://<our hostname>/app/apm","accept-encoding":"gzip, deflate, br","accept-language":"sl,en;q=0.9,en-GB;q=0.8,en-US;q=0.7"},"remoteAddress":"192.168.101.91","userAgent":"192.168.101.91","referer":"https://<our hostname>/app/apm"},"res":{"statusCode":400,"responseTime":30038,"contentLength":9},"message":"GET /api/apm/settings/agent-configuration/services 400 30038ms - 9.0B"}
kibana | {"type":"response","@timestamp":"2020-01-23T09:15:47Z","tags":["access:apm"],"pid":7,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"<our hostname>:443","user-agent":"Go-http-client/1.1","content-length":"65","accept":"application/json","content-type":"application/json","kbn-version":"7.3.1","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"172.18.0.1","userAgent":"172.18.0.1"},"res":{"statusCode":404,"responseTime":15,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 15ms - 9.0B"}
kibana | {"type":"response","@timestamp":"2020-01-23T09:15:58Z","tags":["access:apm"],"pid":7,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"<our hostname>:443","user-agent":"Go-http-client/1.1","content-length":"30","accept":"application/json","content-type":"application/json","kbn-version":"7.3.1","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"172.18.0.1","userAgent":"172.18.0.1"},"res":{"statusCode":404,"responseTime":20,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 20ms - 9.0B"}
kibana | {"type":"response","@timestamp":"2020-01-23T09:16:17Z","tags":["access:apm"],"pid":7,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"<our hostname>:443","user-agent":"Go-http-client/1.1","content-length":"65","accept":"application/json","content-type":"application/json","kbn-version":"7.3.1","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"172.18.0.1","userAgent":"172.18.0.1"},"res":{"statusCode":404,"responseTime":20,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 20ms - 9.0B"}
kibana | {"type":"response","@timestamp":"2020-01-23T09:16:25Z","tags":["access:apm"],"pid":7,"method":"get","statusCode":200,"req":{"url":"/api/apm/settings/agent-configuration","method":"get","headers":{"host":"<our hostname>","connection":"keep-alive","sec-fetch-dest":"empty","kbn-version":"7.3.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.53 Safari/537.36 Edg/80.0.361.33","content-type":"application/json","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"https://<our hostname>/app/apm","accept-encoding":"gzip, deflate, br","accept-language":"sl,en;q=0.9,en-GB;q=0.8,en-US;q=0.7"},"remoteAddress":"192.168.101.91","userAgent":"192.168.101.91","referer":"https://<our hostname>/app/apm"},"res":{"statusCode":200,"responseTime":17,"contentLength":9},"message":"GET /api/apm/settings/agent-configuration 200 17ms - 9.0B"}
kibana | {"type":"response","@timestamp":"2020-01-23T09:16:29Z","tags":["access:apm"],"pid":7,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"<our hostname>:443","user-agent":"Go-http-client/1.1","content-length":"30","accept":"application/json","content-type":"application/json","kbn-version":"7.3.1","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"172.18.0.1","userAgent":"172.18.0.1"},"res":{"statusCode":404,"responseTime":17,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 17ms - 9.0B"}
kibana | {"type":"response","@timestamp":"2020-01-23T09:16:47Z","tags":["access:apm"],"pid":7,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"<our hostname>:443","user-agent":"Go-http-client/1.1","content-length":"65","accept":"application/json","content-type":"application/json","kbn-version":"7.3.1","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"172.18.0.1","userAgent":"172.18.0.1"},"res":{"statusCode":404,"responseTime":21,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 21ms - 9.0B"}
kibana | {"type":"response","@timestamp":"2020-01-23T09:16:47Z","tags":["access:apm"],"pid":7,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"<our hostname>:443","user-agent":"Go-http-client/1.1","content-length":"65","accept":"application/json","content-type":"application/json","kbn-version":"7.3.1","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"172.18.0.1","userAgent":"172.18.0.1"},"res":{"statusCode":404,"responseTime":16,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 16ms - 9.0B"}
kibana | Error: Request Timeout after 30000ms
kibana | at /usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:362:15
kibana | at Timeout.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:391:7)
kibana | at ontimeout (timers.js:436:11)
kibana | at tryOnTimeout (timers.js:300:5)
kibana | at listOnTimeout (timers.js:263:5)
kibana | at Timer.processTimers (timers.js:223:10)
kibana | {"type":"response","@timestamp":"2020-01-23T09:16:25Z","tags":["access:apm"],"pid":7,"method":"get","statusCode":400,"req":{"url":"/api/apm/settings/agent-configuration/services","method":"get","headers":{"host":"<our hostname>","connection":"keep-alive","sec-fetch-dest":"empty","kbn-version":"7.3.1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.53 Safari/537.36 Edg/80.0.361.33","content-type":"application/json","accept":"*/*","sec-fetch-site":"same-origin","sec-fetch-mode":"cors","referer":"https://<our hostname>/app/apm","accept-encoding":"gzip, deflate, br","accept-language":"sl,en;q=0.9,en-GB;q=0.8,en-US;q=0.7"},"remoteAddress":"192.168.101.91","userAgent":"192.168.101.91","referer":"https://<our hostname>/app/apm"},"res":{"statusCode":400,"responseTime":30041,"contentLength":9},"message":"GET /api/apm/settings/agent-configuration/services 400 30041ms - 9.0B"}
kibana | {"type":"response","@timestamp":"2020-01-23T09:17:01Z","tags":["access:apm"],"pid":7,"method":"post","statusCode":404,"req":{"url":"/api/apm/settings/agent-configuration/search","method":"post","headers":{"host":"<our hostname>:443","user-agent":"Go-http-client/1.1","content-length":"30","accept":"application/json","content-type":"application/json","kbn-version":"7.3.1","kbn-xsrf":"1","accept-encoding":"gzip"},"remoteAddress":"172.18.0.1","userAgent":"172.18.0.1"},"res":{"statusCode":404,"responseTime":41,"contentLength":9},"message":"POST /api/apm/settings/agent-configuration/search 404 41ms - 9.0B"}
Из вывода это выглядит так Kibana не может найти конфигурацию сервера, но установка была выполнена успешно, но я также не могу получить доступ к настройкам APM, которые были перемещены в Kibana (время ожидания с ошибкой 404).
Другие запросы или запрос из одни и те же показатели работают хорошо без какой-либо увеличенной задержки.
Агент APM является последней версией агента Django (Python). И показатели ресурсов и кластеров, и статус действительно хороши, поэтому кластер должен быть достаточно мощным.
У нас также было тестирование развертывания в Kubernetes, и службы иногда отображаются в интерфейсе APM, но он повернут в настоящее время выключен.
Среда и версии:
Elasticsearch развернут на GCP в Docker контейнерах (4 узла на разных ВМ + 1 ВМ с сервером APM, Kibana и клиентским узлом Elasticsearch).
ES, Kibana, APM-сервер версии: 7.3.1