Проблема дескриптора файла журнала openstf для Linux - PullRequest
Новая
       60

Проблема дескриптора файла журнала openstf для Linux

0 голосов
/ 18 февраля 2019

До недавнего времени я сталкивался с очень своеобразным предупреждением о файле журнала logstash [filewatch.tailmode.handlers.createinitial] open_file OPEN_WARN_INTERVAL is '300', я искал в Интернете любую возможную помощь, чтобы копать его сам, но не смог найти ни одного потока, предлагающего решение по очередиЯ хотел бы опубликовать его здесь с полными деталями:

Одна Ссылка от Google Другая от обсуждения.elastic.co и anothe_one и несколько других.

ФАЙЛ ЛОГА: /var/log/logstash/logstash-plain.log

[2019-02-18T04:04:07,946][WARN ][filewatch.tailmode.handlers.createinitial] open_file OPEN_WARN_INTERVAL is '300'
[2019-02-18T04:04:07,946][WARN ][filewatch.tailmode.handlers.createinitial] open_file OPEN_WARN_INTERVAL is '300'
[2019-02-18T04:04:07,946][WARN ][filewatch.tailmode.handlers.createinitial] open_file OPEN_WARN_INTERVAL is '300'

И СОСТОЯНИЕ: 

[root@sj-logstash ~]# systemctl status logstash -l
● logstash.service - logstash
   Loaded: loaded (/etc/systemd/system/logstash.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/logstash.service.d
           └─logstashlimit.conf
   Active: active (running) since Sun 2019-02-17 11:10:07 PST; 16h ago
 Main PID: 24558 (java)
   CGroup: /system.slice/logstash.service
           └─24558 /bin/java -Xms1g -Xmx1g -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -cp /usr/share/logstash/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/share/logstash/logstash-core/lib/jars/commons-codec-1.11.jar:/usr/share/logstash/logstash-core/lib/jars/commons-compiler-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/error_prone_annotations-2.0.18.jar:/usr/share/logstash/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/guava-22.0.jar:/usr/share/logstash/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-annotations-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-core-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-databind-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.5.jar:/usr/share/logstash/logstash-core/lib/jars/janino-3.0.8.jar:/usr/share/logstash/logstash-core/lib/jars/jruby-complete-9.1.13.0.jar:/usr/share/logstash/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-api-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-core-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/log4j-slf4j-impl-2.9.1.jar:/usr/share/logstash/logstash-core/lib/jars/logstash-core.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/share/logstash/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/share/logstash/logstash-core/lib/jars/slf4j-api-1.7.25.jar org.logstash.Logstash --path.settings /etc/logstash

Feb 18 04:01:40 sj-logstash logstash[24558]: [2019-02-18T04:01:40,279][WARN ][filewatch.tailmode.handlers.createinitial] open_file OPEN_WARN_INTERVAL is '300'
Feb 18 04:01:40 sj-logstash logstash[24558]: [2019-02-18T04:01:40,279][WARN ][filewatch.tailmode.handlers.createinitial] open_file OPEN_WARN_INTERVAL is '300'
Feb 18 04:01:40 sj-logstash logstash[24558]: [2019-02-18T04:01:40,279][WARN ][filewatch.tailmode.handlers.createinitial] open_file OPEN_WARN_INTERVAL is '300'
Feb 18 04:01:40 sj-logstash logstash[24558]: [2019-02-18T04:01:40,280][WARN ][filewatch.tailmode.handlers.createinitial] open_file OPEN_WARN_INTERVAL is '300'

УСТАНОВКА ФАЙЛА ОТКРЫТОГО ФАЙЛА LOGSTASH:

Поскольку я использую RHEL 7.6, я создал файл ниже, чтобы установить лимит открытия файла для logstash. 

[root@sj-logstash ~]# cat /etc/systemd/system/logstash.service.d/logstashlimit.conf
[Service]
LimitNOFILE=65535

Ниже файла, который я уже видел, я только что установил лимит: 

[root@sj-logstash ~]# cat /etc/systemd/system/logstash.service
[Unit]
Description=logstash

[Service]
Type=simple
User=logstash
Group=logstash
# Load env vars from /etc/default/ and /etc/sysconfig/ if they exist.
# Prefixing the path with '-' makes it try to load, but if the file doesn't
# exist, it continues onward.
EnvironmentFile=-/etc/default/logstash
EnvironmentFile=-/etc/sysconfig/logstash
ExecStart=/usr/share/logstash/bin/logstash "--path.settings" "/etc/logstash"
Restart=always
WorkingDirectory=/
Nice=19
LimitNOFILE=65535 

[Install]
WantedBy=multi-user.target

УРОВЕНЬ СИСТЕМЫ Настройки дескриптора файла: 

[root@sj-logstash ~]# cat /etc/sysctl.conf
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
vm.max_map_count = 262144


[root@sj-logstash ~]# cat /etc/security/limits.conf 

*               soft    nofile 1024
*               hard    nofile 65535 
root            soft    nofile 1024
root            hard    nofile 65535

Настройки JVM LOGSTASH: 

[root@elasticS0104 logstash]# cat /etc/logstash/jvm.options
-Xms7g
-Xmx7g
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djruby.compile.invokedynamic=true
-Djruby.jit.threshold=0
-XX:+HeapDumpOnOutOfMemoryError
-Djava.security.egd=file:/dev/urandom

ВХОД В LOGSTASH Конфигурации: 

[root@sj-logstash ~]# cd  /etc/logstash/conf.d/
[root@sj-logstash conf.d]# ls -l
total 8
-rw-r--r-- 1 root root  917 Feb 17 00:46 syslog.conf
-rw-r--r-- 1 root root 1003 Feb  7 02:14 rmlogs.conf

Logstash config для syslog: 

[root@sj-logstash conf.d]# cat dpc-syslog.conf
input {
  file {
    path => [ "/data_elk/SYSTEMS/*.log" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    max_open_files => 65535 
    type => "dpc-syslog"
  }
}

filter {
  if [type] == "dpc-syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp } %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => [ "@version", "host", "path", "messages" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
 }
}
}
output {
        if [type] == "dpc-syslog" {
        elasticsearch {
                hosts => "elasticS0101:9200"
                manage_template => false
                index => "dpc-syslog-%{+YYYY.MM.dd}"
                document_type => "messages"
  }
 }
}

Logstash config для других пользовательских журналов: 

[root@sj-logstash conf.d]# cat rmlogs.conf
input {
  file {
    path => [ "/data_elk/rmlogs/*.txt" ]
    start_position => beginning
    sincedb_path => "/dev/null"
    max_open_files => 65535 
    type => "rmlog"
  }
}

filter {
  if [type] == "rmlog" {
    grok {
     match => { "message" => "%{HOSTNAME:Hostname},%{DATE:Date},%{HOUR:dt_h}:%{MINUTE:dt_m},%{NUMBER:duration}-%{WORD:hm},%{USER:User},%{USER:User_1} %{NUMBER:Pid} %{NUMBER:float} %{NUMBER:float} %{NUMBER:Num_1} %{NUMBER:Num_2} %{DATA} (?:%{HOUR:dt_h1}:|)(?:%{MINUTE:dt_m1}|) (?:%{HOUR:dt_h2}:|)(?:%{MINUTE:dt_m2}|)%{GREEDYDATA:CMD},%{GREEDYDATA:PWD_PATH}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      remove_field => [ "path","minute3","minute2","host","hour2","hour3","Num_1","Num_2","message" ]
      remove_tag => ["_grokparsefailure"]
   }
 }
}

output {
        if [type] == "rmlog" {
        elasticsearch {
                hosts => "elasticS0101:9200"
                manage_template => false
                index => "dpc-rmlog-%{+YYYY.MM.dd}"
  }
 }
}

LOGSTASH logstash.yml file: 

# cat /etc/logstash/logstash.yml | grep -v "#"
path.data: /var/lib/logstash


path.logs: /var/log/logstash

Буду признателен за любую помощь.

...