Hadoop-Streaming: сокращение задачи в состоянии ожидания говорит: «Нет места для сокращения задачи». - PullRequest
0 голосов
/ 01 ноября 2011

Моя задача карты успешно завершена, и я вижу журналы приложений, но редуктор остается в состоянии ожидания

Kind    % Complete  Num Tasks   Pending Running Complete    Killed  Failed/Killed Task Attempts
map        100.00%  200 0   0   200 0   0 / 40
reduce  0.00%   1   1   0   0   0   0 / 0  

Когда я смотрю на задачу сокращения, я вижу

All Task Attempts

No Task Attempts found

Когда я вижу hadoop-hduser-jobtracker-master.log, я вижу следующее:

2011-10-31 00:00:00,238 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67730714624 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:01,634 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 68041449472 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:03,863 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397398528 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:04,532 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67729956864 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:04,695 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67998732288 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:06,865 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:08,057 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67948941312 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:09,566 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67728035840 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:11,118 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67907883008 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:12,258 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397365760 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:14,119 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67867385856 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:16,061 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:17,120 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67813683200 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:19,386 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67726835712 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:20,122 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 67896156160 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:21,949 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397390336 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:23,124 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_master:localhost/127.0.0.1:43078 has 68150616064 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:25,078 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker1:localhost/127.0.0.1:34794 has 67397365760 bytes free; but we expect reduce input to take 184825113602
2011-10-31 00:00:25,762 WARN org.apache.hadoop.mapred.JobInProgress: No room for reduce task. Node tracker_worker2:localhost/127.0.0.1:35984 has 67725717504 bytes free; but we expect reduce input to take 184825113602

Как мне исправить эту проблему?

Пожалуйста, помогите

1 Ответ

1 голос
/ 01 ноября 2011

Ваш диск на хосте, на котором запущен редуктор, слишком мал. Ваш вход редуктора занимает ок. 172 ГБ, но у вас есть только 63 ГБ бесплатно.

Возможно, объединитель может быть полезен для вас. Посмотрите здесь: http://tumblr.mobocracy.net/post/6589143971/using-a-combiner-with-hadoop-jobs

Добро пожаловать на сайт PullRequest, где вы можете задавать вопросы и получать ответы от других членов сообщества.
...