Glusterfs не может создать том - PullRequest
0 голосов
/ 12 марта 2020

Мой glusterfs был установлен в k8s, используя heketi. Раньше все работало нормально, но сегодня я вдруг не смог создать объем. Ошибка выглядит следующим образом:

[root@localhost grafana]# heketi-cli -s http://10.10.11.181:8080 --user admin --secret runsdata-gf-admin volume create --size=1 --replica=3 --clusters=8fecf14f0a49ea787aa5a35816237cd1 --name=runsdata-monitoring-grafana-data
Error: volume start: runsdata-monitoring-grafana-data: failed: Commit failed on 172.16.5.152. Please check log file for details.

, затем я проверяю журнал pod glusterfs, часть журнала, как показано ниже:

[2020-03-12 07:04:34.004415] I [rpc-clnt.c:1014:rpc_clnt_connection_init] 0-management: setting frame-timeout to 600
[2020-03-12 07:04:34.340139] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped 
[2020-03-12 07:04:34.340226] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: nfs service is stopped 
[2020-03-12 07:04:34.340265] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed 
[2020-03-12 07:04:34.340833] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped 
[2020-03-12 07:04:34.340882] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped 
[2020-03-12 07:04:34.368301] I [MSGID: 106618] [glusterd-svc-helper.c:901:glusterd_attach_svc] 0-glusterd: adding svc glustershd (volume=runsdata-monitoring-grafana-data) to existing process with pid 198 
[2020-03-12 07:04:34.368344] I [MSGID: 106498] [glusterd-svc-helper.c:747:__glusterd_send_svc_configure_req] 0-management: not connected yet 
The message "I [MSGID: 106498] [glusterd-svc-helper.c:747:__glusterd_send_svc_configure_req] 0-management: not connected yet" repeated 14 times between [2020-03-12 07:04:34.368344] and [2020-03-12 07:04:48.370243]
[2020-03-12 07:04:49.370380] W [MSGID: 106617] [glusterd-svc-helper.c:932:glusterd_attach_svc] 0-glusterd: attach failed for glustershd(volume=runsdata-monitoring-grafana-data) 
[2020-03-12 07:04:49.370747] E [MSGID: 106048] [glusterd-shd-svc.c:483:glusterd_shdsvc_start] 0-glusterd: Failed to attach shd svc(volume=runsdata-monitoring-grafana-data) to pid=198 
[2020-03-12 07:04:49.371163] E [MSGID: 106122] [glusterd-mgmt.c:326:gd_mgmt_v3_commit_fn] 0-management: Volume start commit failed. 
[2020-03-12 07:04:49.371193] E [MSGID: 106122] [glusterd-mgmt.c:1582:glusterd_mgmt_v3_commit] 0-management: Commit failed for operation Start on local node 
[2020-03-12 07:04:49.371217] E [MSGID: 106122] [glusterd-mgmt.c:2336:glusterd_mgmt_v3_initiate_all_phases] 0-management: Commit Op Failed 
[2020-03-12 07:04:49.623179] I [run.c:242:runner_log] (-->/usr/lib64/glusterfs/7.1/xlator/mgmt/glusterd.so(+0x3c63d) [0x7f64aec6e63d] -->/usr/lib64/glusterfs/7.1/xlator/mgmt/glusterd.so(+0xe39d5) [0x7f64aed159d5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f64b49a8025] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/stop/pre/S29CTDB-teardown.sh --volname=runsdata-monitoring-grafana-data --last=no 
[2020-03-12 07:04:49.632675] E [run.c:242:runner_log] (-->/usr/lib64/glusterfs/7.1/xlator/mgmt/glusterd.so(+0x3c63d) [0x7f64aec6e63d] -->/usr/lib64/glusterfs/7.1/xlator/mgmt/glusterd.so(+0xe3933) [0x7f64aed15933] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7f64b49a8025] ) 0-management: Failed to execute script: /var/lib/glusterd/hooks/1/stop/pre/S30samba-stop.sh --volname=runsdata-monitoring-grafana-data --last=no 
[2020-03-12 07:04:49.632877] I [MSGID: 106542] [glusterd-utils.c:8659:glusterd_brick_signal] 0-glusterd: sending signal 15 to brick with pid 2505 
[2020-03-12 07:04:50.633158] I [socket.c:864:__socket_shutdown] 0-management: intentional socket shutdown(20)
[2020-03-12 07:04:50.770813] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: nfs already stopped 
[2020-03-12 07:04:50.770958] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: nfs service is stopped 
[2020-03-12 07:04:50.771054] I [MSGID: 106599] [glusterd-nfs-svc.c:81:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed 
[2020-03-12 07:04:50.771705] I [MSGID: 106131] [glusterd-proc-mgmt.c:86:glusterd_proc_stop] 0-management: bitd already stopped 
[2020-03-12 07:04:50.771790] I [MSGID: 106568] [glusterd-svc-mgmt.c:265:glusterd_svc_stop] 0-management: bitd service is stopped 
[2020-03-12 07:04:50.804019] I [MSGID: 106619] [glusterd-svc-helper.c:954:glusterd_detach_svc] 0-management: removing svc glustershd (volume=runsdata-monitoring-grafana-data) from existing process with pid 198

Я не могу видеть из журнала ошибок, что именно вызвал сбой.

Другая информация: У каждого узла моего кластерного кластера все еще есть пространство 4G в VG. Самовосстановление (172.16.5.152) не в сети. Как сделать это онлайн?

...