metrics-сервер с CrashLoopBackOff - PullRequest
       44

metrics-сервер с CrashLoopBackOff

1 голос
/ 31 января 2020

Я не могу запустить модуль метрики-сервера, возникает ошибка crashloopbackoff. Запустив "kubectl get pods -n kube-system", я вижу следующий вывод:

NAME READY STATUS RESTARTS AGE
calico-kube-controllers-6b9d4c8765-mht4w 1/1 Running 0 17m
calico-node-2rmzg 1/1 Running 0 17m
coredns-6955765f44-87kbn 1/1 Running 0 57m
coredns-6955765f44-wzrrt 1/1 Running 0 57m
etcd-master-node 1/1 Running 0 56m
kube-apiserver-master-node 1/1 Running 0 56m
kube-controller-manager-master-node 1/1 Running 0 56m
kube-flannel-ds-amd64-6htmh 1/1 Running 0 28m
kube-proxy-xvksz 1/1 Running 0 57m
kube-scheduler-master-node 1/1 Running 0 56m
metrics-server-7d9ffcffbd-r4gf9 0/1 CrashLoopBackOff 7 16m

Затем, запустив "kubectl description pods metrics-server -n kube-system", я вижу следующее:

Name: metrics-server-7d9ffcffbd-r4gf9
Namespace: kube-system
Priority: 0
Node: master-node/10.221.194.166
Start Time: Thu, 30 Jan 2020 17:07:37 -0300
Labels: k8s-app=metrics-server
pod-template-hash=7d9ffcffbd
Annotations: cni.projectcalico.org/podIP: 192.168.77.133/32
Status: Running
IP: 192.168.77.133
IPs:
IP: 192.168.77.133
Controlled By: ReplicaSet/metrics-server-7d9ffcffbd
Containers:
metrics-server:
Container ID: docker://90ccb84ccf10b130ac93620d105d80c244208b8753c48bb498d646cd3e0c5c17
Image: k8s.gcr.io/metrics-server-amd64:v0.3.6
Image ID: docker-pullable://k8s.gcr.io/metrics-server-amd64@sha256:c9c4e95068b51d6b33a9dccc61875df07dc650abbf4ac1a19d58b4628f89288b
Port: 4443/TCP
Host Port: 0/TCP
Command:
/metrics-server
--kubelet-insecure-tls
--kubelet-preferred-address-types=InternalIP
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 2
Started: Thu, 30 Jan 2020 17:24:10 -0300
Finished: Thu, 30 Jan 2020 17:24:11 -0300
Ready: False
Restart Count: 8
Environment:
Mounts:
/tmp from tmp-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from metrics-server-token-rxfq2 (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
tmp-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit:
metrics-server-token-rxfq2:
Type: Secret (a volume populated by a Secret)
SecretName: metrics-server-token-rxfq2
Optional: false
QoS Class: BestEffort
Node-Selectors: beta.kubernetes.io/os=linux
kubernetes.io/arch=amd64
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message

Normal Scheduled 19m default-scheduler Successfully assigned kube-system/metrics-server-7d9ffcffbd-r4gf9 to master-node
Normal Pulling 18m (x4 over 19m) kubelet, master-node Pulling image "k8s.gcr.io/metrics-server-amd64:v0.3.6"
Normal Pulled 18m (x4 over 19m) kubelet, master-node Successfully pulled image "k8s.gcr.io/metrics-server-amd64:v0.3.6"
Normal Created 18m (x4 over 19m) kubelet, master-node Created container metrics-server
Normal Started 18m (x4 over 19m) kubelet, master-node Started container metrics-server
Warning BackOff 4m5s (x72 over 19m) kubelet, master-node Back-off restarting failed container

Может кто-нибудь помочь мне разобраться в чем дело? Я видел много подобных случаев, но ни один из них не работал. Кроме того, я запускаю все внутри главного узла.

РЕДАКТИРОВАТЬ 1: Публикация вывода журнала:

Error: error creating self-signed certificates: mkdir apiserver.local.config: read-only file system
Usage:
   [flags]

Flags:
      --alsologtostderr                                         log to standard error as well as files
      --authentication-kubeconfig string                        kubeconfig file pointing at the 'core' kubernetes server with enough rights to create tokenaccessreviews.authentication.k8s.io.
      --authentication-skip-lookup                              If false, the authentication-kubeconfig will be used to lookup missing authentication configuration from the cluster.
      --authentication-token-webhook-cache-ttl duration         The duration to cache responses from the webhook token authenticator. (default 10s)
      --authentication-tolerate-lookup-failure                  If true, failures to look up missing authentication configuration from the cluster are not considered fatal. Note that this can result in authentication that treats all requests as anonymous.
      --authorization-always-allow-paths strings                A list of HTTP paths to skip during authorization, i.e. these are authorized without contacting the 'core' kubernetes server.
      --authorization-kubeconfig string                         kubeconfig file pointing at the 'core' kubernetes server with enough rights to create subjectaccessreviews.authorization.k8s.io.
      --authorization-webhook-cache-authorized-ttl duration     The duration to cache 'authorized' responses from the webhook authorizer. (default 10s)
      --authorization-webhook-cache-unauthorized-ttl duration   The duration to cache 'unauthorized' responses from the webhook authorizer. (default 10s)
      --bind-address ip                                         The IP address on which to listen for the --secure-port port. The associated interface(s) must be reachable by the rest of the cluster, and by CLI/web clients. If blank, all interfaces will be used (0.0.0.0 for all IPv4 interfaces and :: for all IPv6 interfaces). (default 0.0.0.0)
      --cert-dir string                                         The directory where the TLS certs are located. If --tls-cert-file and --tls-private-key-file are provided, this flag will be ignored. (default "apiserver.local.config/certificates")
      --client-ca-file string                                   If set, any request presenting a client certificate signed by one of the authorities in the client-ca-file is authenticated with an identity corresponding to the CommonName of the client certificate.
      --contention-profiling                                    Enable lock contention profiling, if profiling is enabled
  -h, --help                                                    help for this command
      --http2-max-streams-per-connection int                    The limit that the server gives to clients for the maximum number of streams in an HTTP/2 connection. Zero means to use golang's default.
      --kubeconfig string                                       The path to the kubeconfig used to connect to the Kubernetes API server and the Kubelets (defaults to in-cluster config)
      --kubelet-certificate-authority string                    Path to the CA to use to validate the Kubelet's serving certificates.
      --kubelet-insecure-tls                                    Do not verify CA of serving certificates presented by Kubelets.  For testing purposes only.
      --kubelet-port int                                        The port to use to connect to Kubelets. (default 10250)
      --kubelet-preferred-address-types strings                 The priority of node address types to use when determining which address to use to connect to a particular node (default [Hostname,InternalDNS,InternalIP,ExternalDNS,ExternalIP])
      --log-flush-frequency duration                            Maximum number of seconds between log flushes (default 5s)
      --log_backtrace_at traceLocation                          when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                                          If non-empty, write log files in this directory
      --log_file string                                         If non-empty, use this log file
      --logtostderr                                             log to standard error instead of files (default true)
      --metric-resolution duration                              The resolution at which metrics-server will retain metrics. (default 1m0s)
      --profiling                                               Enable profiling via web interface host:port/debug/pprof/ (default true)
      --requestheader-allowed-names strings                     List of client certificate common names to allow to provide usernames in headers specified by --requestheader-username-headers. If empty, any client certificate validated by the authorities in --requestheader-client-ca-file is allowed.
      --requestheader-client-ca-file string                     Root certificate bundle to use to verify client certificates on incoming requests before trusting usernames in headers specified by --requestheader-username-headers. WARNING: generally do not depend on authorization being already done for incoming requests.
      --requestheader-extra-headers-prefix strings              List of request header prefixes to inspect. X-Remote-Extra- is suggested. (default [x-remote-extra-])
      --requestheader-group-headers strings                     List of request headers to inspect for groups. X-Remote-Group is suggested. (default [x-remote-group])
      --requestheader-username-headers strings                  List of request headers to inspect for usernames. X-Remote-User is common. (default [x-remote-user])
      --secure-port int                                         The port on which to serve HTTPS with authentication and authorization.If 0, don't serve HTTPS at all. (default 443)
      --skip_headers                                            If true, avoid header prefixes in the log messages
      --stderrthreshold severity                                logs at or above this threshold go to stderr
      --tls-cert-file string                                    File containing the default x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). If HTTPS serving is enabled, and --tls-cert-file and --tls-private-key-file are not provided, a self-signed certificate and key are generated for the public address and saved to the directory specified by --cert-dir.
      --tls-cipher-suites strings                               Comma-separated list of cipher suites for the server. If omitted, the default Go cipher suites will be use.  Possible values: TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_ECDSA_WITH_RC4_128_SHA,TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_RC4_128_SHA,TLS_RSA_WITH_3DES_EDE_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA,TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_GCM_SHA384,TLS_RSA_WITH_RC4_128_SHA
      --tls-min-version string                                  Minimum TLS version supported. Possible values: VersionTLS10, VersionTLS11, VersionTLS12
      --tls-private-key-file string                             File containing the default x509 private key matching --tls-cert-file.
      --tls-sni-cert-key namedCertKey                           A pair of x509 certificate and private key file paths, optionally suffixed with a list of domain patterns which are fully qualified domain names, possibly with prefixed wildcard segments. If no domain patterns are provided, the names of the certificate are extracted. Non-wildcard matches trump over wildcard matches, explicit domain patterns trump over extracted names. For multiple key/certificate pairs, use the --tls-sni-cert-key multiple times. Examples: "example.crt,example.key" or "foo.crt,foo.key:*.foo.com,foo.com". (default [])
  -v, --v Level                                                 number for the log level verbosity
      --vmodule moduleSpec                                      comma-separated list of pattern=N settings for file-filtered logging

panic: error creating self-signed certificates: mkdir apiserver.local.config: read-only file system

goroutine 1 [running]:
main.main()
    /go/src/github.com/kubernetes-incubator/metrics-server/cmd/metrics-server/metrics-server.go:39 +0x13b

Заранее спасибо:)

1 Ответ

0 голосов
/ 31 января 2020

Смотрит мне проблему с контейнером (или) с главным узлом. Можете ли вы проверить журналы развертывания?

Предполагая, что развертывание относится к этой версии: (https://github.com/kubernetes/kubernetes/blob/master/cluster/addons/metrics-server/metrics-server-deployment.yaml)

Мониторинг журналов, таких как:

$ kubectl -n kube-system logs -f deployment/metrics-server-v0.3.6 --all-containers=true --since=10m

в качестве альтернативы вы можете проверить журналы каждого контейнера и контейнера.

$ kubectl logs metrics-server-7d9ffcffbd-r4gf9 -n kube-system -c metrics-server
$ kubectl logs metrics-server-7d9ffcffbd-r4gf9 -n kube-system -c metrics-server-nanny
...