Поскольку проблема с дисковым пространством на узле Harbour, мы увеличиваем его и перезапускаем процесс, но GUI не работает должным образом для таких действий, как репликация, политика хранения, сканирование CVE
Я проверил журналы на стороне сервера :
[root@ip-10-0-25-133 ec2-user]# docker container ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2abf5d05dab7 goharbor/nginx-photon:v1.9.3 "nginx -g 'daemon of…" 6 months ago Up 6 months (healthy) 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp nginx
f33407aa2177 goharbor/harbor-jobservice:v1.9.3 "/harbor/harbor_jobs…" 6 months ago Up 7 weeks (healthy) harbor-jobservice
6420cc8f84ab goharbor/notary-server-photon:v0.6.1-v1.9.3 "/bin/sh -c 'migrate…" 6 months ago Up 6 months notary-server
2546c8894905 goharbor/harbor-core:v1.9.3 "/harbor/harbor_core" 6 months ago Up 6 months (healthy) harbor-core
c04e08177850 goharbor/clair-photon:v2.1.0-v1.9.3 "/docker-entrypoint.…" 6 months ago Up 6 months (healthy) 6060-6061/tcp clair
5e318f8c61d5 goharbor/notary-signer-photon:v0.6.1-v1.9.3 "/bin/sh -c 'migrate…" 6 months ago Up 6 months notary-signer
2555d6e8ff9e goharbor/chartmuseum-photon:v0.9.0-v1.9.3 "/docker-entrypoint.…" 6 months ago Up 6 months (healthy) 9999/tcp chartmuseum
599417cd4413 goharbor/harbor-registryctl:v1.9.3 "/harbor/start.sh" 6 months ago Up 6 months (healthy) registryctl
8967774fa4f2 goharbor/redis-photon:v1.9.3 "redis-server /etc/r…" 6 months ago Up 6 months (healthy) 6379/tcp redis
c4181096df06 goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.3 "/entrypoint.sh /etc…" 6 months ago Up 6 months (healthy) 5000/tcp registry
28583420d59b goharbor/harbor-db:v1.9.3 "/docker-entrypoint.…" 6 months ago Up 6 months (healthy) 5432/tcp harbor-db
f06fa3ac558d goharbor/harbor-portal:v1.9.3 "nginx -g 'daemon of…" 6 months ago Up 6 months (healthy) 8080/tcp harbor-portal
a358bc227c1c goharbor/harbor-log:v1.9.3 "/bin/sh -c /usr/loc…" 6 months ago Up 6 months (healthy) 127.0.0.1:1514->10514/tcp harbor-log
[root@ip-10-0-25-133 ec2-user]# docker logs harbor-log
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
Имеется ошибка rsyslogd, связанная с гаванью, которая, я полагаю, заблокировала предыдущее действие на GUI.
Попытка перезапустить контейнеры через docker compose не помогает, 2 контейнера остаются в неработоспособном состоянии:
[root@ip-10-0-25-133 harbor]# docker-compose start
Starting log ... done
Starting registry ... done
Starting registryctl ... done
Starting postgresql ... done
Starting portal ... done
Starting redis ... done
Starting core ... done
Starting jobservice ... done
Starting proxy ... done
Starting notary-signer ... done
Starting notary-server ... done
Starting clair ... done
Starting chartmuseum ... done
[root@ip-10-0-25-133 harbor]# docker container ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
2abf5d05dab7 goharbor/nginx-photon:v1.9.3 "nginx -g 'daemon of…" 7 months ago Up 3 seconds (health: starting) 0.0.0.0:4443->4443/tcp, 0.0.0.0:80->8080/tcp, 0.0.0.0:443->8443/tcp nginx
f33407aa2177 goharbor/harbor-jobservice:v1.9.3 "/harbor/harbor_jobs…" 7 months ago Up 3 seconds (health: starting) harbor-jobservice
6420cc8f84ab goharbor/notary-server-photon:v0.6.1-v1.9.3 "/bin/sh -c 'migrate…" 7 months ago Up 4 seconds notary-server
2546c8894905 goharbor/harbor-core:v1.9.3 "/harbor/harbor_core" 7 months ago Up 2 seconds (health: starting) harbor-core
c04e08177850 goharbor/clair-photon:v2.1.0-v1.9.3 "/docker-entrypoint.…" 7 months ago Up 3 seconds (health: starting) 6060-6061/tcp clair
5e318f8c61d5 goharbor/notary-signer-photon:v0.6.1-v1.9.3 "/bin/sh -c 'migrate…" 7 months ago Up 5 seconds notary-signer
2555d6e8ff9e goharbor/chartmuseum-photon:v0.9.0-v1.9.3 "/docker-entrypoint.…" 7 months ago Up 8 seconds (health: starting) 9999/tcp chartmuseum
599417cd4413 goharbor/harbor-registryctl:v1.9.3 "/harbor/start.sh" 7 months ago Up 7 seconds (health: starting) registryctl
8967774fa4f2 goharbor/redis-photon:v1.9.3 "redis-server /etc/r…" 7 months ago Up 7 seconds (health: starting) 6379/tcp redis
c4181096df06 goharbor/registry-photon:v2.7.1-patch-2819-2553-v1.9.3 "/entrypoint.sh /etc…" 7 months ago Up 7 seconds (health: starting) 5000/tcp registry
28583420d59b goharbor/harbor-db:v1.9.3 "/docker-entrypoint.…" 7 months ago Up 7 seconds (health: starting) 5432/tcp harbor-db
f06fa3ac558d goharbor/harbor-portal:v1.9.3 "nginx -g 'daemon of…" 7 months ago Up 7 seconds (health: starting) 8080/tcp harbor-portal
a358bc227c1c goharbor/harbor-log:v1.9.3 "/bin/sh -c /usr/loc…" 7 months ago Up 9 seconds (health: starting) 127.0.0.1:1514->10514/tcp harbor-log
Журналы гавани снова указывают на проблему дискового пространства:
[root@ip-10-0-25-133 harbor]# docker logs harbor-jobservice
Error response from daemon: configured logging driver does not support reading
[root@ip-10-0-25-133 harbor]# docker logs clair
Error response from daemon: configured logging driver does not support reading
[root@ip-10-0-25-133 harbor]# docker logs harbor-log
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: file '/var/log/docker/registry.log'[18] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.1910.0 try https://www.rsyslog.com/e/2027 ]
Дисковое пространство не заполнено на хосте docker:
[root@ip-10-0-25-133 harbor]# df -ha
Filesystem Size Used Avail Use% Mounted on
sysfs 0 0 0 - /sys
proc 0 0 0 - /proc
devtmpfs 3.8G 0 3.8G 0% /dev
securityfs 0 0 0 - /sys/kernel/security
tmpfs 3.9G 0 3.9G 0% /dev/shm
devpts 0 0 0 - /dev/pts
tmpfs 3.9G 383M 3.5G 10% /run
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
cgroup 0 0 0 - /sys/fs/cgroup/systemd
pstore 0 0 0 - /sys/fs/pstore
bpf 0 0 0 - /sys/fs/bpf
cgroup 0 0 0 - /sys/fs/cgroup/devices
cgroup 0 0 0 - /sys/fs/cgroup/freezer
cgroup 0 0 0 - /sys/fs/cgroup/pids
cgroup 0 0 0 - /sys/fs/cgroup/net_cls,net_prio
cgroup 0 0 0 - /sys/fs/cgroup/hugetlb
cgroup 0 0 0 - /sys/fs/cgroup/cpuset
cgroup 0 0 0 - /sys/fs/cgroup/perf_event
cgroup 0 0 0 - /sys/fs/cgroup/memory
cgroup 0 0 0 - /sys/fs/cgroup/cpu,cpuacct
cgroup 0 0 0 - /sys/fs/cgroup/blkio
cgroup 0 0 0 - /sys/fs/cgroup/rdma
configfs 0 0 0 - /sys/kernel/config
/dev/xvda2 489G 383G 107G 79% /
selinuxfs 0 0 0 - /sys/fs/selinux
systemd-1 - - - - /proc/sys/fs/binfmt_misc
hugetlbfs 0 0 0 - /dev/hugepages
debugfs 0 0 0 - /sys/kernel/debug
mqueue 0 0 0 - /dev/mqueue
binfmt_misc 0 0 0 - /proc/sys/fs/binfmt_misc
tracefs 0 0 0 - /sys/kernel/debug/tracing
tmpfs 782M 0 782M 0% /run/user/1000
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/cac923a265f3adc2ba8a8b3f14a0fb21330db4a73562a484475cc782deb000fb/merged
shm 64M 0 64M 0% /var/lib/docker/containers/a358bc227c1cc3f986255adc03d354b773fb18f7a383182632b303e8977ddc0a/mounts/shm
nsfs 0 0 0 - /run/docker/netns/b0b97ca08f9a
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/b8f287f1437be444902922a600eb4ae1fb83c9de1be9d439f9d36a08995ff544/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/6db8129dde48fcf90260c06d287e2e02db335dc80d097d8166b92ef0c1105e4e/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/69d464cb11c9c12dd754190f734eaabbed2fe867e5cfa76d7812a068e0801602/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/b2177176bd4c93cf12ff717aabe563c8469cf6bec426ca17e8d82df0618a3b0e/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/11b8dd2688d07ad2a1b81fe11f235f7d3e648580c32c7757109d15fad8aa9030/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/8b0fe0a0b616f6d2343a919e62590b62b0bb714fbce4373936d5a3962d0cb12c/merged
shm 64M 0 64M 0% /var/lib/docker/containers/599417cd44133ac65ed8f084e1c645bf04b41548fc71cf150aa3e2ef224df7a8/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/2555d6e8ff9e3586a41f6f37fc66244d0b3496187da724787cfac175c7f999c4/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/f06fa3ac558da59cf8be1387b811e6d4e622d1ea49a4d97277eb68edf05e6388/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/8967774fa4f28c3df91ca40722dad81949b1d6c2fe60e3bbcf083cf325ebd7f2/mounts/shm
nsfs 0 0 0 - /run/docker/netns/a6281418be81
nsfs 0 0 0 - /run/docker/netns/053f162aea2a
nsfs 0 0 0 - /run/docker/netns/2c01de1d2aa5
shm 64M 4.0K 64M 1% /var/lib/docker/containers/28583420d59b3e4316b72f7a4d56d5ddf5fdf8c5932860f1f017d7b47ce2b0cf/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c4181096df06edcddf3b2406556523ef26a47c7f441382bcd42446802cb708ec/mounts/shm
nsfs 0 0 0 - /run/docker/netns/19276a843cd7
nsfs 0 0 0 - /run/docker/netns/8d1949d48460
nsfs 0 0 0 - /run/docker/netns/85ba684c4adf
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/7faa4fa7c2fa7247d4f7a889f7d47ac97aa5002d83a03fc28bfa48e951e860ad/merged
shm 64M 0 64M 0% /var/lib/docker/containers/5e318f8c61d509346c688b4a3514597f571654620e4bd04dac536caaaa41ddf5/mounts/shm
nsfs 0 0 0 - /run/docker/netns/f8e546c2d757
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/d93e3d145924701e719d7a5fd9e93531c36b46407052e14ec578ca887678f94d/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/92736d5ae32243282bb26ebf9410f645388e5b8b1c1d498c369ad56282211dc6/merged
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/19653cd43524e1094144b7daaeb55b3305f688c1d64bd6eacdd899f2b7c57c38/merged
shm 64M 0 64M 0% /var/lib/docker/containers/6420cc8f84ab45b145e6f142154c3ad3c09a3aa69455ded75fb33aab3415183c/mounts/shm
nsfs 0 0 0 - /run/docker/netns/c31bdc8081b5
shm 64M 0 64M 0% /var/lib/docker/containers/f33407aa21776052609f2edaec1f0c637069992279cc9d19968f2e2d77b2d965/mounts/shm
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/6f53a0994736d049eebd058f2f188e606acfa753e428e605a22943074eca7428/merged
nsfs 0 0 0 - /run/docker/netns/02436d61c648
shm 64M 0 64M 0% /var/lib/docker/containers/2abf5d05dab7d47f8fc528f78d112f7a0c3404d87b60649e897f349439e86c58/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/c04e081778502e8b8e0137512f6d9275daefa2d3641ce220f6d1f7e29a40b378/mounts/shm
overlay 489G 383G 107G 79% /var/lib/docker/overlay2/63e4dbe4d6213f6dcc1322f3691c7ff3244fb71e6562888b955d6bc4f93c74a4/merged
shm 64M 0 64M 0% /var/lib/docker/containers/2546c88949055855eef6967c63365b97e06ca38f46c781a336e2e6d7e67226bf/mounts/shm
nsfs 0 0 0 - /run/docker/netns/9f232ba1bf77
nsfs 0 0 0 - /run/docker/netns/1f50ca91cfa6
nsfs 0 0 0 - /run/docker/netns/2a0c5ac700e3
[root@ip-10-0-25-133 harbor]#
Это не помогает: https://www.rsyslog.com/solving-rsyslog-write-errors/