Я новичок в настройке Graphite, однако я установил графит на 2 блоках с балансировщиком нагрузки, и теперь я мог видеть, что график часто ломается на графитовой веб-странице. Когда я вошел в 2 ящика и мог заметить, что кэш углерода часто перезапускался и видел несколько журналов ошибок. Помогите мне разобраться в проблеме и исправить ее.
Состояние углеродного кэша ::::
carbon_aggregator RUNNING pid 12, uptime 3 days, 19:23:18
carbon_cache RUNNING pid 10580, uptime 1:03:17 <---- restarted
carbon_relay RUNNING pid 2558, uptime 1 day, 22:35:31
collectd RUNNING pid 13, uptime 3 days, 19:23:18
cron RUNNING pid 8, uptime 3 days, 19:23:18
graphite_webapp RUNNING pid 3083, uptime 1 day, 4:53:55
nginx RUNNING pid 16, uptime 3 days, 19:23:18
sshd FATAL Exited too quickly (process log may have details)
syslog-ng RUNNING pid 18, uptime 3 days, 19:23:18
Журнал углеродного кэша
2020 04:12:22 :: [creates] new metric prod.eastus.supply-ump-1.app.10-48-241-6.com.company.ump.workers.templateselector.SupplyTemplateSele
ctorFactory.supplyTemplatesPerRequest.BANNER.23.count matched schema default_1min_for_1day
28/01/2020 04:12:22 :: [creates] new metric prod.eastus.supply-ump-1.app.10-48-241-6.com.company.ump.workers.templateselector.SupplyTemplateSele
ctorFactory.supplyTemplatesPerRequest.BANNER.23.count matched aggregation schema default
28/01/2020 04:12:22 :: [creates] creating database file /opt/graphite/storage/whisper/prod/eastus/supply-ump-1/app/10-48-241-6/com/company/ump/w
orkers/templateselector/SupplyTemplateSelectorFactory/supplyTemplatesPerRequest/BANNER/23/count.wsp (archive=[(60, 30240), (300, 25920), (3600,
21600)] xff=None agg=None)
28/01/2020 04:20:12 :: [query] 127.0.0.1:36472 disconnected
28/01/2020 04:20:12 :: [query] 127.0.0.1:36468 disconnected
28/01/2020 04:20:12 :: [query] 127.0.0.1:36456 disconnected
28/01/2020 04:20:12 :: [query] 127.0.0.1:36482 disconnected
28/01/2020 04:20:23 :: [query] 127.0.0.1:43628 connected
28/01/2020 04:20:33 :: [query] 127.0.0.1:43646 connected
28/01/2020 04:20:45 :: [query] 127.0.0.1:43684 connected
28/01/2020 04:22:34 :: [query] 127.0.0.1:43878 connected
28/01/2020 04:41:39 :: [query] 127.0.0.1:43646 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:43878 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:43628 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:43684 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:42160 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:41992 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:41962 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:41984 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:36464 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:36480 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:36562 disconnected
28/01/2020 04:41:40 :: [query] 127.0.0.1:36560 disconnected
28/01/2020 04:42:03 :: [query] 127.0.0.1:45442 connected
28/01/2020 04:42:03 :: [query] 127.0.0.1:45444 connected
28/01/2020 04:42:03 :: [query] 127.0.0.1:45446 connected
28/01/2020 04:42:03 :: [query] 127.0.0.1:45448 connected
28/01/2020 04:42:04 :: [query] 127.0.0.1:45452 connected
28/01/2020 04:42:04 :: [query] 127.0.0.1:45456 connected
28/01/2020 04:42:05 :: [query] 127.0.0.1:45458 connected
28/01/2020 04:42:05 :: [query] 127.0.0.1:45460 connected
28/01/2020 04:42:07 :: [query] 127.0.0.1:45462 connected
28/01/2020 04:46:00 :: [query] 127.0.0.1:45442 disconnected
28/01/2020 04:46:00 :: [query] 127.0.0.1:45448 disconnected
28/01/2020 04:46:00 :: [query] 127.0.0.1:45462 disconnected
28/01/2020 04:46:00 :: [query] 127.0.0.1:45452 disconnected
28/01/2020 04:46:00 :: [query] 127.0.0.1:45456 disconnected
28/01/2020 04:46:00 :: [query] 127.0.0.1:45444 disconnected
28/01/2020 04:46:20 :: [query] 127.0.0.1:45458 disconnected
28/01/2020 04:46:20 :: [query] 127.0.0.1:45460 disconnected
28/01/2020 04:46:20 :: [query] 127.0.0.1:45446 disconnected
28/01/2020 04:47:27 :: [query] 127.0.0.1:45788 connected
28/01/2020 04:47:27 :: [query] 127.0.0.1:45790 connected
28/01/2020 04:47:32 :: [query] 127.0.0.1:45794 connected
28/01/2020 04:47:50 :: [query] 127.0.0.1:45806 connected
28/01/2020 04:48:22 :: [query] 127.0.0.1:45832 connected
28/01/2020 04:48:22 :: [query] 127.0.0.1:45830 connected
28/01/2020 04:48:57 :: [query] 127.0.0.1:45858 connected
28/01/2020 04:48:58 :: [query] 127.0.0.1:45858 disconnected
28/01/2020 04:48:58 :: [query] 127.0.0.1:45790 disconnected
28/01/2020 04:48:58 :: [query] 127.0.0.1:45806 disconnected
28/01/2020 04:49:09 :: [query] 127.0.0.1:45794 disconnected
28/01/2020 04:49:48 :: [query] 127.0.0.1:45832 connection lost: Connection to the other side was lost in a non-clean fashion.
28/01/2020 04:49:48 :: [query] 127.0.0.1:45830 connection lost: Connection to the other side was lost in a non-clean fashion.
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/twisted/python/usage.py", line 382, in <lambda>
fn = lambda name, value, m=method: m(value)
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 508, in opt_reactor
installReactor(shortName)
File "/usr/local/lib/python2.7/dist-packages/twisted/application/reactors.py", line 79, in installReactor
for installer in getReactorTypes():
File "/usr/local/lib/python2.7/dist-packages/twisted/plugin.py", line 209, in getPlugins
allDropins = getCache(package)
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-packages/twisted/plugin.py", line 167, in getCache
provider = pluginModule.load()
File "/usr/local/lib/python2.7/dist-packages/twisted/python/modules.py", line 383, in load
return self.pathEntry.pythonPath.moduleLoader(self.name)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/reflect.py", line 464, in namedAny
topLevelPackage = _importAndCheckStack(trialname)
File "/usr/local/lib/python2.7/dist-packages/twisted/plugins/twisted_core.py", line 6, in <module>
from twisted.internet.endpoints import (
exceptions.ImportError: cannot import name _SystemdParser
Unhandled Error
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 595, in parseOptions
usage.Options.parseOptions(self, options)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/usage.py", line 226, in parseOptions
for (cmd, short, parser, doc) in self.subCommands:
File "/usr/local/lib/python2.7/dist-packages/twisted/application/app.py", line 607, in subCommands
for plug in plugins:
File "/usr/local/lib/python2.7/dist-packages/twisted/plugin.py", line 209, in getPlugins
allDropins = getCache(package)
--- <exception caught here> ---
File "/usr/local/lib/python2.7/dist-packages/twisted/plugin.py", line 167, in getCache
provider = pluginModule.load()
File "/usr/local/lib/python2.7/dist-packages/twisted/python/modules.py", line 383, in load
return self.pathEntry.pythonPath.moduleLoader(self.name)
File "/usr/local/lib/python2.7/dist-packages/twisted/python/reflect.py", line 464, in namedAny
topLevelPackage = _importAndCheckStack(trialname)
File "/usr/local/lib/python2.7/dist-packages/twisted/plugins/twisted_core.py", line 6, in <module>
from twisted.internet.endpoints import (
exceptions.ImportError: cannot import name _SystemdParser
Removing stale pidfile /tmp/carbon-cache-a.pid
Starting carbon-cache (instance a)
28/01/2020 05:00:08 :: [console] Log opened.
28/01/2020 05:00:08 :: [console] twistd 11.0.0 (/usr/bin/python 2.7.6) starting up.
28/01/2020 05:00:08 :: [console] reactor class: twisted.internet.epollreactor.EPollReactor.
28/01/2020 05:00:08 :: [console] twisted.internet.protocol.ServerFactory starting on 2478
28/01/2020 05:00:08 :: [console] Starting factory <twisted.internet.protocol.ServerFactory instance at 0x7fa77903d680>
28/01/2020 05:00:08 :: [console] twisted.internet.protocol.ServerFactory starting on 2578
28/01/2020 05:00:08 :: [console] Starting factory <twisted.internet.protocol.ServerFactory instance at 0x7fa77908eb00>
28/01/2020 05:00:08 :: [console] twisted.internet.protocol.ServerFactory starting on 7138
28/01/2020 05:00:08 :: [console] Starting factory <twisted.internet.protocol.ServerFactory instance at 0x7fa77908e8c0>
28/01/2020 05:00:12 :: [listener] MetricPickleReceiver connection with 127.0.0.1:48278 established
28/01/2020 05:00:23 :: [query] 127.0.0.1:46222 connected
28/01/2020 05:00:28 :: [query] 127.0.0.1:46228 connected
Углерод-агрегаторные бревна
28/01/2020 01:58:55 :: [console] <twisted.internet.tcp.Connector instance at 0x7fc16d985290> will retry in 3 seconds
28/01/2020 01:58:55 :: [clients] CarbonClientFactory(127.0.0.1:2578:None)::clientConnectionLost (127.0.0.1:2578) Connection to the other side was lost in a non-clean fashion.
28/01/2020 01:58:55 :: [console] Stopping factory CarbonClientFactory(127.0.0.1:2578:None)
28/01/2020 01:58:59 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2578:None)
28/01/2020 01:58:59 :: [clients] CarbonClientFactory(127.0.0.1:2578:None)::startedConnecting (127.0.0.1:2578)
28/01/2020 01:58:59 :: [clients] CarbonClientProtocol(127.0.0.1:2578:None)::connectionMade
28/01/2020 03:24:53 :: [clients] CarbonClientProtocol(127.0.0.1:2578:None)::connectionLost Connection to the other side was lost in a non-clean fashion.
28/01/2020 03:24:53 :: [console] <twisted.internet.tcp.Connector instance at 0x7fc16d985290> will retry in 4 seconds
28/01/2020 03:24:53 :: [clients] CarbonClientFactory(127.0.0.1:2578:None)::clientConnectionLost (127.0.0.1:2578) Connection to the other side was lost in a non-clean fashion.
28/01/2020 03:24:53 :: [console] Stopping factory CarbonClientFactory(127.0.0.1:2578:None)
28/01/2020 03:24:58 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2578:None)
28/01/2020 03:24:58 :: [clients] CarbonClientFactory(127.0.0.1:2578:None)::startedConnecting (127.0.0.1:2578)
28/01/2020 03:24:58 :: [clients] CarbonClientProtocol(127.0.0.1:2578:None)::connectionMade
28/01/2020 05:00:07 :: [clients] CarbonClientProtocol(127.0.0.1:2578:None)::connectionLost Connection was closed cleanly.
28/01/2020 05:00:07 :: [console] <twisted.internet.tcp.Connector instance at 0x7fc16d985290> will retry in 4 seconds
28/01/2020 05:00:07 :: [clients] CarbonClientFactory(127.0.0.1:2578:None)::clientConnectionLost (127.0.0.1:2578) Connection was closed cleanly.
28/01/2020 05:00:07 :: [console] Stopping factory CarbonClientFactory(127.0.0.1:2578:None)
28/01/2020 05:00:12 :: [console] Starting factory CarbonClientFactory(127.0.0.1:2578:None)
28/01/2020 05:00:12 :: [clients] CarbonClientFactory(127.0.0.1:2578:None)::startedConnecting (127.0.0.1:2578)
28/01/2020 05:00:12 :: [clients] CarbonClientProtocol(127.0.0.1:2578:None)::connectionMade