Проблема синтаксического анализа Fluentbit Docker Журналы (Systemd) для сообщения GELF Вывод в Kubernetes - PullRequest
Новая
       106

Проблема синтаксического анализа Fluentbit Docker Журналы (Systemd) для сообщения GELF Вывод в Kubernetes

0 голосов
/ 22 февраля 2020

Я получаю Docker (Systemd) журналы и пытаюсь отправить его в формате GELF на выход Graylog 3, но журнал не в правильном формате, и Graylog сбрасывает его.

Я следую это ссылки:

См. Мой стек ниже:

INPUT

  • Docker версия 1.13.1
    • Docker Формат журнала = > JSON
    • Docker Драйвер журнала => Journald => systemd
  • Fluent-бит 1.3, работающий в качестве демона в Kubernetes
  • Kubernetes 1.17
  • ОС Хост: CentOS 7

ВЫХОД

  • Формат вывода сообщения: GELF 1.1
  • Централизованный журнал => Graylog 3

Мои конфигурации Kubernetes:

fluent-bit-configmap.yaml 

apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
  namespace: log
  labels:
    k8s-app: fluent-bit
data:
  # Configuration files: server, input, filters and output
  # ======================================================
  fluent-bit.conf: |
    [SERVICE]
      Flush         1
      Log_Level     info
      Daemon        off
      Parsers_File  parser-docker.conf

    [INPUT]
      Name                    tail
      Tag                     kube.*
      Path                    /var/log/messages
      Parser                  docker
      DB                      /var/log/flb_kube.db
      Mem_Buf_Limit           5MB
      Refresh_Interval        10

    [FILTER]
      Name                    kubernetes
      Match                   kube.*
      Merge_Log_Key           log
      Merge_Log               On
      Keep_Log                Off
      Annotations             Off
      Labels                  Off

    [FILTER]
      Name                    nest
      Match                   *
      Operation               lift
      Nested_under            log

    [OUTPUT]
      Name                    gelf
      Match                   kube.*
      Host                    10.142.15.214
      Port                    12201
      Mode                    tcp
      Gelf_Short_Message_Key  data

    [OUTPUT]
      Name   stdout
      Match  *

  parser-docker.conf: |
    [PARSER]
      Name                    docker
      Format                  json
      Time_Key                time
      Time_Format             %Y-%m-%dT%H:%M:%S.%L
      Time_Keep               Off

fluent-bit-ds.yaml 

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: log
  labels:
    k8s-app: fluent-bit-logging
    version: v1
    kubernetes.io/cluster-service: "true"
spec:
  selector:
    matchLabels:
      k8s-app: fluent-bit-logging
      version: v1
      kubernetes.io/cluster-service: "true"
  template:
    metadata:
      labels:
        k8s-app: fluent-bit-logging
        version: v1
        kubernetes.io/cluster-service: "true"
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "2020"
        prometheus.io/path: /api/v1/metrics/prometheus
        fluentbit.io/exclude: "true"
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:1.3.5
        imagePullPolicy: Always
        ports:
          - containerPort: 2020
        env:
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: systemdlog
          mountPath: /run/log
        - name: fluent-bit-config
          mountPath: /fluent-bit/etc/
      terminationGracePeriodSeconds: 10
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: systemdlog
        hostPath:
          path: /run/log
      - name: fluent-bit-config
        configMap:
          name: fluent-bit-config
      serviceAccountName: fluent-bit
      tolerations:
      - key: node-role.kubernetes.io/master
        operator: Exists
        effect: NoSchedule
      - operator: "Exists"
        effect: "NoExecute"
      - operator: "Exists"
        effect: "NoSchedule"

fluent -bit-role-binding.yaml 

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: fluent-bit-read
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluent-bit-read
subjects:
- kind: ServiceAccount
  name: fluent-bit
  namespace: log

fluent-bit-role.yaml 

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: fluent-bit-read
rules:
- apiGroups: [""]
  resources:
  - namespaces
  - pods
  verbs: ["get", "list", "watch"]

fluent-bit-service-account. yaml 

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluent-bit
  namespace: log

My Fluentbit OUTPUT (STDOUT) только для отладки: 

$ kubectl logs -f fluent-bit-2bzxb -n log


[9] host.docker.service: [1582317069.020005000, {"PRIORITY"=>"6", "_TRANSPORT"=>"journal", "_PID"=>"1486", "_UID"=>"0", "_GID"=>"0", "_COMM"=>"dockerd-current", "_EXE"=>"/usr/bin/dockerd-current", "_CMDLINE"=>"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2", "_CAP_EFFECTIVE"=>"1fffffffff", "_SYSTEMD_CGROUP"=>"/system.slice/docker.service", "_SYSTEMD_UNIT"=>"docker.service", "_SYSTEMD_SLICE"=>"system.slice", "_BOOT_ID"=>"18d81c7e97e6419f999af50af13060c8", "_MACHINE_ID"=>"b5447343d5617cb5f7fd428164927298", "_HOSTNAME"=>"k8s-worker-1", "CONTAINER_TAG"=>"d72b414a9bcc", "CONTAINER_ID"=>"d72b414a9bcc", "CONTAINER_ID_FULL"=>"d72b414a9bccef31dbaa4c8473b06d63583195ebde9a8b729a06a81b68233144", "CONTAINER_NAME"=>"k8s_fluent-bit_fluent-bit-zg7pz_log_43007dd0-ce10-4c6d-a97f-e7369f866879_0", "MESSAGE"=>"
[9] host.docker.service: [1582317068.864240000, {"_TRANSPORT"=>"journal", "_PID"=>"1486", "_UID"=>"0", "_GID"=>"0", "_COMM"=>"dockerd-current", "_EXE"=>"/usr/bin/dockerd-current", "_CMDLINE"=>"/usr/bin/dockerd-current --add-runtime docker-runc=/usr/libexec/docker/docker-runc-current --default-runtime=docker-runc --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/libexec/docker/docker-init-current --seccomp-profile=/etc/docker/seccomp.json --selinux-enabled --log-driver=journald --signature-verification=false --storage-driver overlay2", "_CAP_EFFECTIVE"=>"1fffffffff", "_SYSTEMD_CGROUP"=>"/system.slice/docker.service", "_SYSTEMD_UNIT"=>"docker.service", "_SYSTEMD_SLICE"=>"system.slice", "_BOOT_ID"=>"18d81c7e97e6419f999af50af13060c8", "_MACHINE_ID"=>"b5447343d5617cb5f7fd428164927298", "_HOSTNAME"=>"k8s-worker-1", "PRIORITY"=>"3", "CONTAINER_TAG"=>"d3383915a884", "CONTAINER_ID"=>"d3383915a884", "CONTAINER_ID_FULL"=>"d3383915a884fb0e2b40189e7db1a1131161e57dba39a40824accf1b2aa59f22", "CONTAINER_NAME"=>"k8s_demo-app_demo-app-6c79ffd869-trstt_default_cd6c5f4d-ec44-4f43-9b92-d7bb28a5f676_1", "MESSAGE"=>"2020/02/21 20:31:08 10.142.15.231:48012 GET /", "_SOURCE_REALTIME_TIMESTAMP"=>"1582317068863691"}]", "_SOURCE_REALTIME_TIMESTAMP"=>"1582317069000509"}]"

Проблема заключается в том, как правильно отформатировать журнал в формате GELF для отправки в Graylog 3

...