Redis с Docker не хранит вакансии Sidekiq - PullRequest
Новая
       17

Redis с Docker не хранит вакансии Sidekiq

0 голосов
/ 19 мая 2019

У меня есть приложение Rails, и я хотел его докертизировать. Я использую Digital Ocean, и там установлены Docker (18.09.4) и docker compose (1.18.0). Мой файл docker-compose.yml выглядит следующим образом: 

version: '2'
services:
  app:
    build:
      context: .
      dockerfile: ./docker/app/Dockerfile
    depends_on:
      - db
      - redis
    command: bundle exec puma -C config/puma.rb
    expose:
      - 3000
    env_file: .env
    volumes:
      - .:/app
      - /home/deploy/apps/myapp/shared/public/assets:/var/www/myapp/public/assets

  cron:
    build:
      context: .
      dockerfile: ./docker/app/Dockerfile
    command: bash -c 'bundle exec whenever -i && cron -f'
    depends_on:
      - db
      - redis
    volumes:
      - .:/app
    env_file: .env

  db:
    image: postgres
    volumes:
      - postgres_data:/var/lib/postgresql/data
    env_file: .env

  redis:
    image: redis:4.0.14
    command: redis-server --appendonly yes
    ports:
      - 6379:6379
    volumes:
      - redis_data:/data

  sidekiq_default:
    depends_on:
      - db
      - redis
    build:
      context: .
      dockerfile: ./docker/app/Dockerfile
    command: bundle exec sidekiq -C config/sidekiq_default.yml
    volumes:
      - .:/app
    env_file: .env

  sidekiq_import:
    depends_on:
      - db
      - redis
    build:
      context: .
      dockerfile: ./docker/app/Dockerfile
    command: bundle exec sidekiq -C config/sidekiq_import.yml
    volumes:
      - .:/app
    env_file: .env

  web:
    build:
      context: .
      dockerfile: ./docker/web/Dockerfile
    depends_on:
      - app
    command: nginx -g 'daemon off;'
    ports:
      - 80:80

volumes:
  postgres_data:
  redis_data:

В чем проблема? Глядя на интерфейс Sidekiq, я время от времени теряю работу. Я даже сделал redis.set("foo", "bar") на следующий день после его возвращения nil. Похоже, что контейнер перезапускается, даже если состояние контейнера, например, up X days. Когда я посмотрел журналы, я получил следующий вывод: 

25229:C 18 May 21:15:52.497 # Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): Permission denied
1:M 18 May 21:15:52.597 # Background AOF rewrite terminated with error
1:M 18 May 21:15:52.697 * Starting automatic rewriting of AOF on 119964390% growth
1:M 18 May 21:15:52.697 * Background append only file rewriting started by pid 25230
25230:C 18 May 21:15:52.698 # Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): Permission denied
1:M 18 May 21:15:52.798 # Background AOF rewrite terminated with error
1:M 18 May 21:15:52.898 * Starting automatic rewriting of AOF on 119964390% growth
1:M 18 May 21:15:52.898 * Background append only file rewriting started by pid 25231
25231:C 18 May 21:15:52.899 # Opening the temp file for AOF rewrite in rewriteAppendOnlyFile(): Permission denied
1:M 18 May 21:15:52.999 # Background AOF rewrite terminated with error
1:signal-handler (1558214153) Received SIGTERM scheduling shutdown...
1:M 18 May 21:15:53.099 # User requested shutdown...
1:M 18 May 21:15:53.099 * Calling fsync() on the AOF file.
1:M 18 May 21:15:53.099 # Redis is now ready to exit, bye bye...
1:C 18 May 21:15:57.373 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 18 May 21:15:57.373 # Redis version=4.0.14, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 18 May 21:15:57.373 # Configuration loaded
1:M 18 May 21:15:57.378 * Running mode=standalone, port=6379.
1:M 18 May 21:15:57.379 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 18 May 21:15:57.379 # Server initialized
1:M 18 May 21:15:57.379 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 18 May 21:16:36.388 * DB loaded from append only file: 39.008 seconds
1:M 18 May 21:16:36.389 * Ready to accept connections
1:M 18 May 22:58:03.031 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 18 May 22:59:43.054 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 18 May 23:01:37.063 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 18 May 23:10:19.038 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 18 May 23:13:17.090 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 18 May 23:16:09.021 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
1:M 19 May 03:25:37.328 * DB saved on disk
1:M 19 May 03:25:37.337 * DB saved on disk

Так что это говорит о том, что есть проблема с AOF. Как я могу это исправить? Что-то не так в конфигурации docker-compose?

Edit: Я перезапустил контейнер и продолжал смотреть журналы и рабочие места. Через некоторое время задания снова исчезли и в логах контейнера ничего не осталось: 

1:C 19 May 12:29:46.531 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 19 May 12:29:46.532 # Redis version=4.0.14, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 19 May 12:29:46.532 # Configuration loaded
1:M 19 May 12:29:46.538 * Running mode=standalone, port=6379.
1:M 19 May 12:29:46.539 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 19 May 12:29:46.539 # Server initialized
1:M 19 May 12:29:46.539 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 19 May 12:30:56.916 * DB loaded from append only file: 70.377 seconds
1:M 19 May 12:30:56.917 * Ready to accept connections
1:M 19 May 13:44:02.620 * DB saved on disk
1:M 19 May 13:44:02.625 * DB saved on disk
1:signal-handler (1558274030) Received SIGTERM scheduling shutdown...
1:M 19 May 13:53:50.487 # User requested shutdown...
1:M 19 May 13:53:50.487 * Calling fsync() on the AOF file.
1:M 19 May 13:53:50.487 # Redis is now ready to exit, bye bye...
1:C 19 May 13:53:51.422 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
1:C 19 May 13:53:51.422 # Redis version=4.0.14, bits=64, commit=00000000, modified=0, pid=1, just started
1:C 19 May 13:53:51.422 # Configuration loaded
1:M 19 May 13:53:51.424 * Running mode=standalone, port=6379.
1:M 19 May 13:53:51.424 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
1:M 19 May 13:53:51.424 # Server initialized
1:M 19 May 13:53:51.424 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
1:M 19 May 13:54:35.554 * DB loaded from append only file: 44.130 seconds
1:M 19 May 13:54:35.554 * Ready to accept connections
...