Прогресс карты Mapreduce внезапно резко упал. Так же, как map100% к карте67% - PullRequest
Новая
       30

Прогресс карты Mapreduce внезапно резко упал. Так же, как map100% к карте67%

0 голосов
/ 08 июля 2019

Я использовал потоковую передачу hadoop для запуска кода mapreduce.

Это мой статус бега 

19/07/08 20:43:32 INFO mapreduce.Job:  map 0% reduce 0%
19/07/08 20:43:44 INFO mapreduce.Job:  map 1% reduce 0%
19/07/08 20:43:47 INFO mapreduce.Job:  map 2% reduce 0%
19/07/08 20:44:00 INFO mapreduce.Job:  map 3% reduce 0%
19/07/08 20:44:12 INFO mapreduce.Job:  map 5% reduce 0%
19/07/08 20:44:24 INFO mapreduce.Job:  map 6% reduce 0%
19/07/08 20:44:36 INFO mapreduce.Job:  map 8% reduce 0%
19/07/08 20:44:48 INFO mapreduce.Job:  map 9% reduce 0%
19/07/08 20:45:00 INFO mapreduce.Job:  map 10% reduce 0%
19/07/08 20:45:12 INFO mapreduce.Job:  map 12% reduce 0%
19/07/08 20:45:27 INFO mapreduce.Job:  map 13% reduce 0%
19/07/08 20:45:39 INFO mapreduce.Job:  map 14% reduce 0%
19/07/08 20:45:51 INFO mapreduce.Job:  map 16% reduce 0%
19/07/08 20:46:04 INFO mapreduce.Job:  map 17% reduce 0%
19/07/08 20:46:19 INFO mapreduce.Job:  map 19% reduce 0%
19/07/08 20:46:31 INFO mapreduce.Job:  map 20% reduce 0%
19/07/08 20:46:43 INFO mapreduce.Job:  map 21% reduce 0%
19/07/08 20:46:55 INFO mapreduce.Job:  map 23% reduce 0%
19/07/08 20:47:04 INFO mapreduce.Job:  map 24% reduce 0%
19/07/08 20:47:16 INFO mapreduce.Job:  map 25% reduce 0%
19/07/08 20:47:28 INFO mapreduce.Job:  map 27% reduce 0%
19/07/08 20:47:40 INFO mapreduce.Job:  map 28% reduce 0%
19/07/08 20:47:52 INFO mapreduce.Job:  map 29% reduce 0%
19/07/08 20:47:55 INFO mapreduce.Job:  map 30% reduce 0%
19/07/08 20:48:05 INFO mapreduce.Job:  map 31% reduce 0%
19/07/08 20:48:17 INFO mapreduce.Job:  map 32% reduce 0%
19/07/08 20:48:26 INFO mapreduce.Job:  map 33% reduce 0%
19/07/08 20:48:29 INFO mapreduce.Job:  map 34% reduce 0%
19/07/08 20:48:41 INFO mapreduce.Job:  map 35% reduce 0%
19/07/08 20:48:53 INFO mapreduce.Job:  map 36% reduce 0%
19/07/08 20:48:56 INFO mapreduce.Job:  map 37% reduce 0%
19/07/08 20:49:05 INFO mapreduce.Job:  map 38% reduce 0%
19/07/08 20:49:17 INFO mapreduce.Job:  map 39% reduce 0%
19/07/08 20:49:29 INFO mapreduce.Job:  map 41% reduce 0%
19/07/08 20:49:38 INFO mapreduce.Job:  map 42% reduce 0%
19/07/08 20:49:47 INFO mapreduce.Job:  map 43% reduce 0%
19/07/08 20:49:53 INFO mapreduce.Job:  map 44% reduce 0%
19/07/08 20:50:02 INFO mapreduce.Job:  map 45% reduce 0%
19/07/08 20:50:14 INFO mapreduce.Job:  map 46% reduce 0%
19/07/08 20:50:20 INFO mapreduce.Job:  map 47% reduce 0%
19/07/08 20:50:23 INFO mapreduce.Job:  map 48% reduce 0%
19/07/08 20:50:35 INFO mapreduce.Job:  map 49% reduce 0%
19/07/08 20:50:48 INFO mapreduce.Job:  map 51% reduce 0%
19/07/08 20:51:00 INFO mapreduce.Job:  map 52% reduce 0%
19/07/08 20:51:13 INFO mapreduce.Job:  map 53% reduce 0%
19/07/08 20:51:25 INFO mapreduce.Job:  map 55% reduce 0%
19/07/08 20:51:34 INFO mapreduce.Job:  map 56% reduce 0%
19/07/08 20:51:46 INFO mapreduce.Job:  map 57% reduce 0%
19/07/08 20:51:55 INFO mapreduce.Job:  map 58% reduce 0%
19/07/08 20:51:58 INFO mapreduce.Job:  map 59% reduce 0%
19/07/08 20:52:10 INFO mapreduce.Job:  map 60% reduce 0%
19/07/08 20:52:22 INFO mapreduce.Job:  map 62% reduce 0%
19/07/08 20:52:34 INFO mapreduce.Job:  map 63% reduce 0%
19/07/08 20:52:46 INFO mapreduce.Job:  map 64% reduce 0%
19/07/08 20:52:58 INFO mapreduce.Job:  map 66% reduce 0%
19/07/08 20:53:10 INFO mapreduce.Job:  map 67% reduce 0%
19/07/08 20:53:22 INFO mapreduce.Job:  map 68% reduce 0%
19/07/08 20:53:25 INFO mapreduce.Job:  map 69% reduce 0%
19/07/08 20:53:28 INFO mapreduce.Job:  map 70% reduce 0%
19/07/08 20:53:31 INFO mapreduce.Job:  map 71% reduce 0%
19/07/08 20:53:37 INFO mapreduce.Job:  map 72% reduce 0%
19/07/08 20:53:40 INFO mapreduce.Job:  map 73% reduce 0%
19/07/08 20:53:43 INFO mapreduce.Job:  map 74% reduce 0%
19/07/08 20:53:46 INFO mapreduce.Job:  map 75% reduce 0%
19/07/08 20:53:52 INFO mapreduce.Job:  map 76% reduce 0%
19/07/08 20:53:55 INFO mapreduce.Job:  map 77% reduce 0%
19/07/08 20:53:58 INFO mapreduce.Job:  map 78% reduce 0%
19/07/08 20:54:01 INFO mapreduce.Job:  map 79% reduce 0%
19/07/08 20:54:07 INFO mapreduce.Job:  map 80% reduce 0%
19/07/08 20:54:10 INFO mapreduce.Job:  map 81% reduce 0%
19/07/08 20:54:13 INFO mapreduce.Job:  map 82% reduce 0%
19/07/08 20:54:16 INFO mapreduce.Job:  map 83% reduce 0%
19/07/08 20:54:22 INFO mapreduce.Job:  map 84% reduce 0%
19/07/08 20:54:26 INFO mapreduce.Job:  map 92% reduce 0%
19/07/08 20:54:29 INFO mapreduce.Job:  map 93% reduce 0%
19/07/08 20:54:32 INFO mapreduce.Job:  map 95% reduce 0%
19/07/08 20:54:35 INFO mapreduce.Job:  map 96% reduce 0%
19/07/08 20:54:38 INFO mapreduce.Job:  map 97% reduce 0%
19/07/08 20:54:41 INFO mapreduce.Job:  map 98% reduce 0%
19/07/08 20:54:44 INFO mapreduce.Job:  map 99% reduce 0%
19/07/08 20:54:47 INFO mapreduce.Job:  map 100% reduce 0%
19/07/08 20:57:15 INFO mapreduce.Job:  map 67% reduce 0%
19/07/08 21:01:42 INFO mapreduce.Job:  map 100% reduce 0%
19/07/08 21:01:59 INFO mapreduce.Job:  map 100% reduce 67%
19/07/08 21:02:17 INFO mapreduce.Job:  map 100% reduce 68%
19/07/08 21:02:32 INFO mapreduce.Job:  map 100% reduce 69%
19/07/08 21:02:47 INFO mapreduce.Job:  map 100% reduce 70%
19/07/08 21:03:05 INFO mapreduce.Job:  map 100% reduce 71%
19/07/08 21:03:20 INFO mapreduce.Job:  map 100% reduce 72%
19/07/08 21:03:39 INFO mapreduce.Job:  map 100% reduce 73%
19/07/08 21:03:54 INFO mapreduce.Job:  map 100% reduce 74%
19/07/08 21:04:06 INFO mapreduce.Job:  map 100% reduce 75%
19/07/08 21:04:18 INFO mapreduce.Job:  map 100% reduce 76%
19/07/08 21:04:33 INFO mapreduce.Job:  map 100% reduce 77%
19/07/08 21:04:49 INFO mapreduce.Job:  map 100% reduce 78%
19/07/08 21:05:01 INFO mapreduce.Job:  map 100% reduce 79%
19/07/08 21:05:13 INFO mapreduce.Job:  map 100% reduce 80%
19/07/08 21:05:31 INFO mapreduce.Job:  map 100% reduce 81%
19/07/08 21:05:46 INFO mapreduce.Job:  map 100% reduce 82%
19/07/08 21:06:01 INFO mapreduce.Job:  map 100% reduce 83%
19/07/08 21:06:20 INFO mapreduce.Job:  map 100% reduce 84%
19/07/08 21:06:35 INFO mapreduce.Job:  map 100% reduce 85%
19/07/08 21:06:53 INFO mapreduce.Job:  map 100% reduce 86%
19/07/08 21:07:08 INFO mapreduce.Job:  map 100% reduce 87%
19/07/08 21:07:23 INFO mapreduce.Job:  map 100% reduce 88%
19/07/08 21:07:42 INFO mapreduce.Job:  map 100% reduce 89%
19/07/08 21:07:57 INFO mapreduce.Job:  map 100% reduce 90%
19/07/08 21:08:12 INFO mapreduce.Job:  map 100% reduce 91%
19/07/08 21:08:30 INFO mapreduce.Job:  map 100% reduce 92%
19/07/08 21:08:45 INFO mapreduce.Job:  map 100% reduce 93%
19/07/08 21:09:03 INFO mapreduce.Job:  map 100% reduce 94%
19/07/08 21:09:18 INFO mapreduce.Job:  map 100% reduce 95%
19/07/08 21:09:36 INFO mapreduce.Job:  map 100% reduce 96%
19/07/08 21:09:51 INFO mapreduce.Job:  map 100% reduce 97%
19/07/08 21:10:10 INFO mapreduce.Job:  map 100% reduce 98%
19/07/08 21:10:25 INFO mapreduce.Job:  map 100% reduce 99%
19/07/08 21:10:43 INFO mapreduce.Job:  map 100% reduce 100%

Map-Reduce Framework
                Map input records=74916938
                Map output records=74916938
                Map output bytes=2818376834
                Map output materialized bytes=879967982
                Input split bytes=132
                Combine input records=0
                Combine output records=0
                Reduce input groups=68739252
                Reduce shuffle bytes=879967982
                Reduce input records=74916938
                Reduce output records=68739252
                Spilled Records=291836337
                Shuffled Maps =1
                Failed Shuffles=0
                Merged Map outputs=1
                GC time elapsed (ms)=2210
                CPU time spent (ms)=1579300
                Physical memory (bytes) snapshot=321605632
                Virtual memory (bytes) snapshot=4165459968
                Total committed heap usage (bytes)=165810176

Это состояние, которое я лично чувствую ненормальным.

19/07/08 20:54:47 INFO mapreduce.Job: карта 100% уменьшить 0%

19/07/08 20:57:15 INFO mapreduce.Job: карта 67% уменьшить 0%

19/07/08 21:01:42 INFO mapreduce.Job: карта 100% уменьшить 0%

Q1: нормально ли переходить со 100% до 67% в статусе карты?

В2. Является ли это статусом карты или статусом понижения, нормально ли переходить от высокого завершения к низкому завершению?

Спасибо за вашу помощь

...