Это мой код или Sidekiq медленный, потому что он не предназначен для доступа к моделям Rails? - PullRequest
Новая
       12

Это мой код или Sidekiq медленный, потому что он не предназначен для доступа к моделям Rails?

0 голосов
/ 03 ноября 2018

Раньше я оборачивал большую часть своего кода внутри Thread.new {}, а затем начинал спотыкаться через Sidekiq, чтобы сделать фоновые задачи немного проще. Спустя несколько недель это разочаровывает, и я не могу понять, так ли это, как я вкладываю код в своих сотрудников sidekiq или в чем заключается сделка.

Например, 

[1] pry(#<DomainWorker>)> domain = Domain.find_by(unique_id: @unique_id)
[2] pry(#<DomainWorker>)> urls
=> [{:url=>"http://admin.example.com/"}, {:url=>"http://example.com/"}, {:url=>"http://www.example.com/"}, {:url=>"https://admin.example.com/"}, {:url=>"https://example.com/"}, {:url=>"https://www.example.com/"}]
[3] pry(#<DomainWorker>)> domain.domain_urls.build(urls)
=> [#<DomainUrl:0x0000557c97f2a3c8 id: nil, domain_id: 32, url: "http://admin.example.com/", created_at: nil, updated_at: nil>,
 #<DomainUrl:0x0000557c97cc6e38 id: nil, domain_id: 32, url: "http://example.com/", created_at: nil, updated_at: nil>,
 #<DomainUrl:0x0000557c97cadd70 id: nil, domain_id: 32, url: "http://www.example.com/", created_at: nil, updated_at: nil>,
 #<DomainUrl:0x0000557c97c9c8e0 id: nil, domain_id: 32, url: "https://admin.example.com/", created_at: nil, updated_at: nil>,
 #<DomainUrl:0x0000557c97c7ed18 id: nil, domain_id: 32, url: "https://example.com/", created_at: nil, updated_at: nil>,
 #<DomainUrl:0x0000557c97c43df8 id: nil, domain_id: 32, url: "https://www.example.com/", created_at: nil, updated_at: nil>]

Чтобы просто запустить команду domain.domain_urls.build(urls), потребовалось около 15 секунд. Если я открою rails c, эта команда запускается мгновенно.

Это противоречит лучшим практикам манипулирования моделями из sidekiq работника или что-то в этом роде? Не уверен, почему иногда это занимает 15-20 секунд, 5 секунд в следующий раз и т. Д. Это очень противоречиво. У меня даже есть MailWorker, который иногда отправляет электронное письмо либо за 2 секунды, либо за 2 минуты. Не уверен, что сделка.

Вот мой config/sidekiq.yml файл конфигурации: 

development:  
  :concurrency: 50
production:  
  :concurrency: 50
:queues:
  - default

Это мой код или просто Sidekiq плохо справляется с этими задачами? Что мне здесь не хватает?

EDIT

Просто отключение eager_load в моей среде разработки, похоже, делает некоторый прогресс.

В config/environments/development.rb:

изменен

config.eager_load = false

до

config.eager_load = true

Он работает некоторое время, но затем снова начинает зависать после того, как куча задач поставлена ​​в очередь и завершена. Теперь он снова застрял.

Если я запускаю show processlist в MySQL, я вижу следующее: 

mysql> show full processlist;
+----+------------+-----------+-------------------+---------+------+----------+-----------------------+
| Id | User       | Host      | db                | Command | Time | State    | Info                  |
+----+------------+-----------+-------------------+---------+------+----------+-----------------------+
| 31 | app_dev | localhost | app_portal_dev | Sleep   | 1017 |          | NULL                  |
| 50 | root       | localhost | NULL              | Query   |    0 | starting | show full processlist |
| 55 | app_dev | localhost | app_portal_dev | Sleep   | 1059 |          | NULL                  |
| 57 | app_dev | localhost | app_portal_dev | Sleep   | 1028 |          | NULL                  |
| 58 | app_dev | localhost | app_portal_dev | Sleep   |  420 |          | NULL                  |
| 61 | app_dev | localhost | app_portal_dev | Sleep   | 1078 |          | NULL                  |
| 62 | app_dev | localhost | app_portal_dev | Sleep   |  425 |          | NULL                  |
| 63 | app_dev | localhost | app_portal_dev | Sleep   |  191 |          | NULL                  |
+----+------------+-----------+-------------------+---------+------+----------+-----------------------+
8 rows in set (0.00 sec)

Кроме того, если я отправлю сигнал уничтожения TTIN работнику Sidekiq, это будет выводом: 

2018-11-03T03:20:29.415Z 65977 TID-gsr3safel WARN: Thread TID-gsr4jqd7l processor
2018-11-03T03:20:29.415Z 65977 TID-gsr3safel WARN: /home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `select'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `rescue in _read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:64:in `_read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:56:in `gets'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:363:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:263:in `block in read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:251:in `io'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:262:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `block in call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:232:in `block (2 levels) in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:370:in `ensure_connected'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:222:in `block in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:307:in `logging'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:221:in `process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:210:in `block in call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:281:in `with_socket_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:209:in `call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1172:in `block in _bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `block in synchronize'
/home/myuser/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1169:in `_bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1214:in `brpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:95:in `block in redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:92:in `redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:89:in `get_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:99:in `fetch'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:82:in `process_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:71:in `run'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:16:in `watchdog'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-11-03T03:20:29.416Z 65977 TID-gsr3safel WARN: Thread TID-gsr4jqddx processor
2018-11-03T03:20:29.416Z 65977 TID-gsr3safel WARN: /home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `select'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `rescue in _read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:64:in `_read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:56:in `gets'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:363:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:263:in `block in read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:251:in `io'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:262:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `block in call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:232:in `block (2 levels) in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:370:in `ensure_connected'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:222:in `block in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:307:in `logging'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:221:in `process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:210:in `block in call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:281:in `with_socket_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:209:in `call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1172:in `block in _bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `block in synchronize'
/home/myuser/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1169:in `_bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1214:in `brpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:95:in `block in redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:92:in `redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:89:in `get_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:99:in `fetch'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:82:in `process_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:71:in `run'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:16:in `watchdog'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-11-03T03:20:29.416Z 65977 TID-gsr3safel WARN: Thread TID-gsr4jqdh1 processor
2018-11-03T03:20:29.416Z 65977 TID-gsr3safel WARN: /home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `select'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `rescue in _read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:64:in `_read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:56:in `gets'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:363:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:263:in `block in read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:251:in `io'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:262:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `block in call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:232:in `block (2 levels) in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:370:in `ensure_connected'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:222:in `block in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:307:in `logging'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:221:in `process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:210:in `block in call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:281:in `with_socket_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:209:in `call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1172:in `block in _bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `block in synchronize'
/home/myuser/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1169:in `_bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1214:in `brpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:95:in `block in redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:92:in `redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:89:in `get_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:99:in `fetch'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:82:in `process_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:71:in `run'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:16:in `watchdog'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-11-03T03:20:29.417Z 65977 TID-gsr3safel WARN: Thread TID-gsr4jqcuh processor
2018-11-03T03:20:29.417Z 65977 TID-gsr3safel WARN: /home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `select'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `rescue in _read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:64:in `_read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:56:in `gets'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:363:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:263:in `block in read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:251:in `io'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:262:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `block in call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:232:in `block (2 levels) in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:370:in `ensure_connected'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:222:in `block in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:307:in `logging'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:221:in `process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:210:in `block in call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:281:in `with_socket_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:209:in `call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1172:in `block in _bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `block in synchronize'
/home/myuser/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1169:in `_bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1214:in `brpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:95:in `block in redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:92:in `redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:89:in `get_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:99:in `fetch'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:82:in `process_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:71:in `run'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:16:in `watchdog'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-11-03T03:20:29.417Z 65977 TID-gsr3safel WARN: Thread TID-gsr4jqd4p processor
2018-11-03T03:20:29.417Z 65977 TID-gsr3safel WARN: /home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `select'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `rescue in _read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:64:in `_read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:56:in `gets'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:363:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:263:in `block in read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:251:in `io'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:262:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `block in call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:232:in `block (2 levels) in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:370:in `ensure_connected'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:222:in `block in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:307:in `logging'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:221:in `process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:210:in `block in call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:281:in `with_socket_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:209:in `call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1172:in `block in _bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `block in synchronize'
/home/myuser/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1169:in `_bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1214:in `brpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:95:in `block in redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:92:in `redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:89:in `get_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:99:in `fetch'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:82:in `process_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:71:in `run'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:16:in `watchdog'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:25:in `block in safe_thread'
2018-11-03T03:20:29.418Z 65977 TID-gsr3safel WARN: Thread TID-gsr4jqcn5 processor
2018-11-03T03:20:29.418Z 65977 TID-gsr3safel WARN: /home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `select'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:68:in `rescue in _read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:64:in `_read_from_socket'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:56:in `gets'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/connection/ruby.rb:363:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:263:in `block in read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:251:in `io'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:262:in `read'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `block in call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:232:in `block (2 levels) in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:370:in `ensure_connected'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:222:in `block in process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:307:in `logging'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:221:in `process'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:118:in `call'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:210:in `block in call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:281:in `with_socket_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis/client.rb:209:in `call_with_timeout'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1172:in `block in _bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `block in synchronize'
/home/myuser/.rvm/rubies/ruby-2.5.1/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:45:in `synchronize'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1169:in `_bpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/redis-4.0.2/lib/redis.rb:1214:in `brpop'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `block in retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:95:in `block in redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:65:in `block (2 levels) in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:64:in `block in with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `handle_interrupt'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/connection_pool-2.2.2/lib/connection_pool.rb:61:in `with'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq.rb:92:in `redis'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/fetch.rb:36:in `retrieve_work'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:89:in `get_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:99:in `fetch'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:82:in `process_one'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/processor.rb:71:in `run'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:16:in `watchdog'
/home/myuser/.rvm/gems/ruby-2.5.1/gems/sidekiq-5.2.2/lib/sidekiq/util.rb:25:in `block in safe_thread'

Ответы [ 2 ]

0 голосов
/ 07 ноября 2018

Кажется, нет ничего плохого в выводе TTIN или в базе данных. Sidekiq идеально подходит для работы с Rails и ActiveRecord. Мы работаем в высоконагруженной производственной среде с около 30 тыс. Одновременных потоков (распределенных между несколькими десятками рабочих серверов), и Sidekiq работает исключительно. Не устанавливайте ненужный высокий параллелизм в режиме разработки, хотя, если вы играете с одной задачей, которая на самом деле не имеет значения.

Попробуйте отладить работника Sidekiq локально - установите точку останова в начале вашего метода и найдите, где она работает медленно. Маловероятно, что сам Sidekiq до этого

0 голосов
/ 04 ноября 2018

Rails режим автозагрузки является однопоточным. Вы можете установить Sidekiq на любой параллелизм, какой захотите, но у вас все еще будет гигантский мьютекс, чтобы однонитировать ваши задания, чтобы они могли безопасно загружаться. Включение eager_load или переключение в рабочий режим отключает автозагрузку.

параллелизм: 50 слишком высоко. Не устанавливайте его выше, чем 25.

Также возможно, что у вас огромное количество потоков, но не одинаковое количество соединений с базой данных, установите pool: 25 в config/database.yml

...