mongodb с PV C на томе NFS, перезапускающемся с "file-rename: rename: Permission denied"

Question

У меня проблемы с моим кластером mongodb в пользовательском кластере Kubernetes. У меня есть том NFS, и поэтому в моем mongodb statefulSet я создаю ClaimTemplate, который обращается к этому хранилищу NFS, чтобы сохранить данные на этом диске.

Работает, я вижу файлы, сохраненные там, но что Я вижу, что мои модули mongodb часто перезагружаются, например 140 раз за 25 дней.

Поэтому я смотрю журналы одного из модулей следующим образом: kubectl logs mongo-0 mongodb --previous, чтобы увидеть журналы перед последний перезапуск.

И каждый раз, когда я вижу эту ошибку здесь:

2020-02-14T08:52:39.133+0000 E STORAGE  [WTCheckpointThread] WiredTiger error (13) [1581670359:133838][1:0x7f6e8137b700], file:WiredTiger.wt, WT_SESSION.checkpoint: /data/db/WiredTiger.turtle.set to /data/db/WiredTiger.turtle: file-rename: rename: Permission denied
2020-02-14T08:52:39.137+0000 E STORAGE  [WTCheckpointThread] WiredTiger error (0) [1581670359:137663][1:0x7f6e8137b700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: encountered an illegal file format or internal value: (__wt_turtle_update, 391)
2020-02-14T08:52:39.137+0000 E STORAGE  [WTCheckpointThread] WiredTiger error (-31804) [1581670359:137717][1:0x7f6e8137b700], file:WiredTiger.wt, WT_SESSION.checkpoint: the process must exit and restart: WT_PANIC: WiredTiger library panic
2020-02-14T08:52:39.137+0000 F -        [WTCheckpointThread] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 366
2020-02-14T08:52:39.137+0000 F -        [WTCheckpointThread] 

***aborting after fassert() failure


2020-02-14T08:52:39.157+0000 F -        [WTCheckpointThread] Got signal: 6 (Aborted).

 0x562363d30251 0x562363d2f469 0x562363d2f94d 0x7f6e86f41890 0x7f6e86bbc067 0x7f6e86bbd448 0x56236248b12e 0x562362558c1e 0x5623625c5f91 0x562362427d13 0x56236242803c 0x56236258c05c 0x562362589bac 0x5623626694bf 0x562362669bff 0x5623625d6306 0x5623625d8049 0x5623625d99b3 0x5623625d9c83 0x5623625bf50a 0x56236253e62f 0x562363c22961 0x562363e3f470 0x7f6e86f3a064 0x7f6e86c6f62d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"562361AF7000","o":"2239251","s":"_ZN5mongo15printStackTraceERSo"},{"b":"562361AF7000","o":"2238469"},{"b":"562361AF7000","o":"223894D"},{"b":"7F6E86F32000","o":"F890"},{"b":"7F6E86B87000","o":"35067","s":"gsignal"},{"b":"7F6E86B87000","o":"36448","s":"abort"},{"b":"562361AF7000","o":"99412E","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"562361AF7000","o":"A61C1E"},{"b":"562361AF7000","o":"ACEF91"},{"b":"562361AF7000","o":"930D13","s":"__wt_err"},{"b":"562361AF7000","o":"93103C","s":"__wt_panic"},{"b":"562361AF7000","o":"A9505C","s":"__wt_turtle_update"},{"b":"562361AF7000","o":"A92BAC","s":"__wt_metadata_update"},{"b":"562361AF7000","o":"B724BF"},{"b":"562361AF7000","o":"B72BFF","s":"__wt_meta_ckptlist_set"},{"b":"562361AF7000","o":"ADF306"},{"b":"562361AF7000","o":"AE1049","s":"__wt_checkpoint"},{"b":"562361AF7000","o":"AE29B3"},{"b":"562361AF7000","o":"AE2C83","s":"__wt_txn_checkpoint"},{"b":"562361AF7000","o":"AC850A"},{"b":"562361AF7000","o":"A4762F","s":"_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv"},{"b":"562361AF7000","o":"212B961","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"562361AF7000","o":"2348470"},{"b":"7F6E86F32000","o":"8064"},{"b":"7F6E86B87000","o":"E862D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.7", "gitVersion" : "2628472127e9f1826e02c665c1d93880a204075e", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-74-generic", "version" : "#84-Ubuntu SMP Thu Dec 19 08:06:28 UTC 2019", "machine" : "x86_64" }, "somap" : [ { "b" : "562361AF7000", "elfType" : 3, "buildId" : "EAE95F4BE97BD38029D21836039E6398277FF9BC" }, { "b" : "7FFE42B82000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "053A6959CC589646998C0B1448414A8AF5B51F67" }, { "b" : "7F6E880D0000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "C0E9A6CE03F960E690EA8F72575FFA29570E4A0B" }, { "b" : "7F6E87CD3000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "CFDB319C26A6DB0ED14D33D44024ED461D8A5C23" }, { "b" : "7F6E87A72000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "90275AC4DD8167F60BC7C599E0DBD63741D8F191" }, { "b" : "7F6E8786E000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "D70B531D672A34D71DB42EB32B68E63F2DCC5B6A" }, { "b" : "7F6E87666000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "A63C95FB33CCA970E141D2E13774B997C1CF0565" }, { "b" : "7F6E87365000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "152C93BA3E8590F7ED0BCDDF868600D55EC4DD6F" }, { "b" : "7F6E8714F000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "BAC839560495859598E8515CBAED73C7799AE1FF" }, { "b" : "7F6E86F32000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "9DA9387A60FFC196AEDB9526275552AFEF499C44" }, { "b" : "7F6E86B87000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "48C48BC6ABB794461B8A558DD76B29876A0551F0" }, { "b" : "7F6E882E7000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "1D98D41FBB1EABA7EC05D0FD7624B85D6F51C03C" } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x562363d30251]
 mongod(+0x2238469) [0x562363d2f469]
 mongod(+0x223894D) [0x562363d2f94d]
 libpthread.so.0(+0xF890) [0x7f6e86f41890]
 libc.so.6(gsignal+0x37) [0x7f6e86bbc067]
 libc.so.6(abort+0x148) [0x7f6e86bbd448]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x56236248b12e]
 mongod(+0xA61C1E) [0x562362558c1e]
 mongod(+0xACEF91) [0x5623625c5f91]
 mongod(__wt_err+0x9D) [0x562362427d13]
 mongod(__wt_panic+0x33) [0x56236242803c]
 mongod(__wt_turtle_update+0x1BC) [0x56236258c05c]
 mongod(__wt_metadata_update+0x1BC) [0x562362589bac]
 mongod(+0xB724BF) [0x5623626694bf]
 mongod(__wt_meta_ckptlist_set+0x29F) [0x562362669bff]
 mongod(+0xADF306) [0x5623625d6306]
 mongod(__wt_checkpoint+0xA9) [0x5623625d8049]
 mongod(+0xAE29B3) [0x5623625d99b3]
 mongod(__wt_txn_checkpoint+0x1C3) [0x5623625d9c83]
 mongod(+0xAC850A) [0x5623625bf50a]
 mongod(_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv+0x23F) [0x56236253e62f]
 mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x131) [0x562363c22961]
 mongod(+0x2348470) [0x562363e3f470]
 libpthread.so.0(+0x8064) [0x7f6e86f3a064]
 libc.so.6(clone+0x6D) [0x7f6e86c6f62d]
-----  END BACKTRACE  -----

Если я, например, подключаюсь к модулю (с помощью команды kubectl exec -it mongo-0 -- /bin/sh) и перечисляю файлы в папке / data / db /, я вижу это:

# ls -la
total 7832895
drwxrwxrwx 2 root root      81920 Feb 14 09:52 .
drwxr-xr-x 4 root root       4096 Sep  5  2018 ..
-rw------- 1 root root         46 Nov 21 08:21 WiredTiger
-rw------- 1 root root         21 Nov 21 08:21 WiredTiger.lock
-rw------- 1 root root       1105 Feb 14 09:52 WiredTiger.turtle
-rw------- 1 root root     737280 Feb 14 09:48 WiredTiger.wt
-rw------- 1 root root       4096 Feb 14 08:52 WiredTigerLAS.wt
-rw------- 1 root root      57344 Feb 14 08:52 _mdb_catalog.wt
-rw------- 1 root root      36864 Feb 14 09:51 collection-0--2261811786273811131.wt
-rw------- 1 root root      36864 Feb 14 09:52 collection-0--3926443192316967741.wt
-rw------- 1 root root     200704 Feb 14 08:53 collection-0-5585625509523565893.wt
-rw------- 1 root root      36864 Feb 14 08:52 collection-0-6343563410859966760.wt
-rw------- 1 root root      16384 Feb 14 08:52 collection-100-6343563410859966760.wt

все файлы управляются root, но в папке все пользователи могут делать что угодно.

Что мне здесь не хватает? почему процесс, который создает все эти файлы, не имеет разрешения на его переименование?

Это определение StatefulSet следующим образом:

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mongo
spec:
  serviceName: "mongo"
  replicas: 3
  template:
    metadata:
      labels:
        app: mongo
    spec:
      containers:
      - name: mongodb
        image: mongo:3-jessie
        command:
        - mongod
        - --replSet
        - rs0
        - --bind_ip_all
        ports:
        - containerPort: 27017
          name: web
        volumeMounts:
        - name: database
          mountPath: /data/db
      - name: init-mongo
        image: mongo:3-jessie
        command:
        - bash
        - /config/init.sh
        volumeMounts:
        - name: config
          mountPath: /config
      volumes:
      - name: config
        configMap:
          name: "mongo-init"
  volumeClaimTemplates:
  - metadata:
      name: database
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: nfs.raid5
      resources:
        requests:
          storage: 30Gi

и это для configMap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: mongo-init
data:
  init.sh: |
    #!/bin/bash
    until /usr/bin/mongo --eval 'printjson(db.serverStatus())'; do
      echo "connecting to local mongo..."
      sleep 2
    done
    echo "connected to local."

    HOST=mongo-0.mongo:27017

    until /usr/bin/mongo --host=${HOST} --eval 'printjson(db.serverStatus())'; do
      echo "connecting to remote mongo..."
      sleep 2
    done
    echo "connected to remote."

    if [[ "${HOSTNAME}" != 'mongo-0' ]]; then
      until /usr/bin/mongo --host=${HOST} --eval="printjson(rs.status())" | grep -v "no replset config has been received"; do
        echo "waiting for replication set initialization"
        sleep 2
      done
      echo "adding self to mongo-0"
      /usr/bin/mongo --host=${HOST} --eval="printjson(rs.add('${HOSTNAME}.mongo'))"
    fi

    if [[ "${HOSTNAME}" == 'mongo-0' ]]; then
      echo "initializing replica set"
      /usr/bin/mongo --eval="printjson(rs.initiate({'_id': 'rs0', 'members': [{'_id': 0, 'host': 'mongo-0.mongo:27017'}]}))"
    fi
    echo "initialized"

    while true; do
      sleep 3600
    done

Это моя конфигурация хранилища NFS, показанная с помощью команды kubectl get pvc -A -o yaml

apiVersion: v1
items:
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-01-13T14:50:11Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: cassandra
    name: cassandra-data-cassandra-0
    namespace: default
    resourceVersion: "59562447"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/cassandra-data-cassandra-0
    uid: ffc81929-3613-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-ffc81929-3613-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 20Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-01-13T14:50:16Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: cassandra
    name: cassandra-data-cassandra-1
    namespace: default
    resourceVersion: "59562488"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/cassandra-data-cassandra-1
    uid: 02c23282-3614-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-02c23282-3614-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 20Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-01-13T14:50:21Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: cassandra
    name: cassandra-data-cassandra-2
    namespace: default
    resourceVersion: "59562528"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/cassandra-data-cassandra-2
    uid: 05d251fc-3614-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 20Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-05d251fc-3614-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 20Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-02-05T10:17:49Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      service: elasticsearch
    name: data-elasticsearch-0
    namespace: default
    resourceVersion: "64027300"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/data-elasticsearch-0
    uid: c2f91498-4800-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 50Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-c2f91498-4800-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 50Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-02-05T10:18:46Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      service: elasticsearch
    name: data-elasticsearch-1
    namespace: default
    resourceVersion: "64027480"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/data-elasticsearch-1
    uid: e4a434c9-4800-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 50Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-e4a434c9-4800-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 50Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-02-05T10:19:46Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      service: elasticsearch
    name: data-elasticsearch-2
    namespace: default
    resourceVersion: "64027629"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/data-elasticsearch-2
    uid: 089e3867-4801-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 50Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-089e3867-4801-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 50Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2019-12-17T10:19:12Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: minio
    name: data-minio-0
    namespace: default
    resourceVersion: "54242369"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/data-minio-0
    uid: ab9f0e15-20b6-11ea-922b-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 30Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-ab9f0e15-20b6-11ea-922b-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 30Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2019-12-17T10:36:28Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: minio
    name: data-minio-1
    namespace: default
    resourceVersion: "54244889"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/data-minio-1
    uid: 152ad7c7-20b9-11ea-922b-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 30Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-152ad7c7-20b9-11ea-922b-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 30Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2019-12-17T10:36:49Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: minio
    name: data-minio-2
    namespace: default
    resourceVersion: "54244969"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/data-minio-2
    uid: 21cdfb6a-20b9-11ea-922b-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 30Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-21cdfb6a-20b9-11ea-922b-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 30Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2019-12-17T10:37:04Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: minio
    name: data-minio-3
    namespace: default
    resourceVersion: "54245023"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/data-minio-3
    uid: 2ade3c07-20b9-11ea-922b-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 30Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-2ade3c07-20b9-11ea-922b-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 30Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-01-20T09:54:32Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: redis
    name: data-redis-0
    namespace: default
    resourceVersion: "60898171"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/data-redis-0
    uid: dbc61d61-3b6a-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-dbc61d61-3b6a-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2019-11-21T08:20:39Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: mongo
    name: database-mongo-0
    namespace: default
    resourceVersion: "49142141"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/database-mongo-0
    uid: cd2bc437-0c37-11ea-922b-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 30Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-cd2bc437-0c37-11ea-922b-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 30Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2019-11-21T08:20:57Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: mongo
    name: database-mongo-1
    namespace: default
    resourceVersion: "49142205"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/database-mongo-1
    uid: d7d7d7c6-0c37-11ea-922b-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 30Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-d7d7d7c6-0c37-11ea-922b-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 30Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2019-11-21T08:21:13Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: mongo
    name: database-mongo-2
    namespace: default
    resourceVersion: "49142269"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/database-mongo-2
    uid: e1add0ee-0c37-11ea-922b-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 30Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-e1add0ee-0c37-11ea-922b-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 30Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-01-14T08:09:50Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: mash-firstupload-protel
    name: storage-mash-firstupload-protel-0
    namespace: default
    resourceVersion: "59703924"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/storage-mash-firstupload-protel-0
    uid: 3cb79ca5-36a5-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-3cb79ca5-36a5-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-01-14T08:09:53Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: mash-firstupload-protel
    name: storage-mash-firstupload-protel-1
    namespace: default
    resourceVersion: "59703971"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/storage-mash-firstupload-protel-1
    uid: 3ec97b97-36a5-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-3ec97b97-36a5-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    phase: Bound
- apiVersion: v1
  kind: PersistentVolumeClaim
  metadata:
    annotations:
      pv.kubernetes.io/bind-completed: "yes"
      pv.kubernetes.io/bound-by-controller: "yes"
      volume.beta.kubernetes.io/storage-provisioner: nfs.raid5
    creationTimestamp: "2020-01-14T08:10:06Z"
    finalizers:
    - kubernetes.io/pvc-protection
    labels:
      app: mash-firstupload-protel
    name: storage-mash-firstupload-protel-2
    namespace: default
    resourceVersion: "59704015"
    selfLink: /api/v1/namespaces/default/persistentvolumeclaims/storage-mash-firstupload-protel-2
    uid: 464dd3d0-36a5-11ea-96eb-00155d01e331
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
    storageClassName: nfs.raid5
    volumeMode: Filesystem
    volumeName: pvc-464dd3d0-36a5-11ea-96eb-00155d01e331
  status:
    accessModes:
    - ReadWriteOnce
    capacity:
      storage: 10Gi
    phase: Bound
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Заранее спасибо за ответы! Если вам нужна какая-либо другая информация, пожалуйста, дайте мне знать, и я отредактирую вопрос.