У меня есть следующие 6 Датододов (дн):
- dn1 - 6 ядер 6 ГБ - 3 слота для карт и 3 слота для уменьшения
- dn2 - 6 ядер 6 ГБ - 3 слота для карт и 3 слота для уменьшения
- dn3 - 6 ядер 6 ГБ - 3 слота для карт и 3 слота для уменьшения
- dn4 - 6 ядер 6 ГБ - 3 слота для карт и 3 слота для уменьшения
- dn5 - 6 ядер 6 ГБ - 3 слота для карт и 3 слота для уменьшения
- dn6 - 6 ядер 6 ГБ - 3 слота для карт и 3 слота для уменьшения
Дело 1 (Доступность)
State of the system
===================
dn1 has 1 mapper running from another job Y; so 2 mapper slots are free
dn2 has 1 mapper running from another job Y; so 2 mapper slots are free
dn3 has 1 mapper running from another job Y; so 2 mapper slots are free
dn4 has 1 mapper running from another job Y; so 2 mapper slots are free
dn5 has 0 mappers running; so 3 mapper slots are free
dn6 has 0 mappers running; so 3 mapper slots are free
State of my input file
======================
I have a file that is distributed in 3 64MB blocks with RF 3 in the following way:
R1(dn1,dn2,dn3)
R2(dn2,dn3,dn4)
R3(dn3,dn4,dn5)
Когда я запускаю задание X для этого файла, необходимо создать 3 сопоставителя, соответствующих 3 блокам данных.
Вопросы
In FIFO: Is Job X still put on the queue waiting for Y to finish considering its a FIFO scheduler and other jobs are running even though "there are other mapper slots free in the same machine" or the FIFO logic kicks in only when no more resources are available in the system and the jobs consequently has to be put on the queue?
In Capacity Scheduler: What would the behavior be?
In Fair Share Scheduler: What would the behavior be?
Случай 2 (дефицит)
State of the system
===================
dn1 has 3 mappers running from another job Y; so 0 mapper slots are free
dn2 has 3 mappers running from another job Y; so 0 mapper slots are free
dn3 has 3 mappers running from another job Y; so 0 mapper slots are free
dn4 has 3 mappers running from another job Y; so 0 mapper slots are free
dn5 has 3 mappers running from another job Y; so 0 mapper slots are free
dn6 has 0 mappers running; so 3 mapper slots are free
У меня есть файл, который распределен в 3 блоках по 64 МБ с RF 3 следующим образом:
R1(dn1,dn2,dn3)
R2(dn2,dn3,dn4)
R3(dn3,dn4,dn5)
Когда я запускаю задание X для этого файла, необходимо создать 3 сопоставителя, соответствующих 3 блокам данных.
Вопросы
Что происходит сейчас:
- Are the 3 mapper tasks created on dn6 (which does not have any of the data blocks of the input file yet) and corresponding data block transferred over the network from say dn1 to dn6?
- If yes, does this same behaviour show in the case of all the three schedulers: FIFO/Capacity/Fair Share?
- If no, then can you elaborate on the behaviour shown for this use case in case of:
- FIFO Scheduler
- Capacity Scheduler
- Fair Share Scheduler