У меня есть приложения spark (2.0.1), развернутые в кластере Hadoop с версией 2.7.0 (я использую HDP 2.5).Все эти искровые приложения работали в обычном режиме с динамическим распределением в течение почти двух лет.Однако на этой неделе произошло исключение.Все эти искровые приложения постоянно запрашивают больше ресурсов, чем они должны быть выделены.
например.со следующей конфигурацией
spark.dynamicAllocation.enabled false
spark.dynamicAllocation.minExecutors 1
spark.dynamicAllocation.maxExecutors 18
spark.dynamicAllocation.executorIdleTimeout 120
Приложение spark продолжает запрашивать более 18 исполнителей (и даже распределяться с большим количеством исполнителей), пока оно не заполнит всю очередь и не получит весь менеджер ресурсов.Этот «симптом» произошел со всеми приложениями spark в моем кластере
Я проверил все связанные журналы: журнал диспетчера ресурсов, журнал приложений и диспетчер узлов, но я не смог найти никаких отклонений.Пожалуйста, помогите мне с этим.
Спасибо за миллион!
Сильфон - это мое вложение для файлов конфигурации
Журнал драйвера искры, который продолжает запрашивать нового исполнителя, пока старые исполнители получаютуничтожается постоянно одновременно
Line 32151: 04/10/2018 22:26:29.616 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 623 has been removed (new total is 27)
Line 32160: 04/10/2018 22:26:31.185 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 650 has registered (new total is 28)
Line 32173: 04/10/2018 22:26:31.384 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 624 has been removed (new total is 27)
Line 32182: 04/10/2018 22:26:33.020 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 651 has registered (new total is 28)
Line 32189: 04/10/2018 22:26:33.999 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 654 has registered (new total is 29)
Line 32205: 04/10/2018 22:26:35.558 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 619 has been removed (new total is 28)
Line 32215: 04/10/2018 22:26:37.608 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 655 has registered (new total is 29)
Line 32218: 04/10/2018 22:26:37.609 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 652 has registered (new total is 30)
Line 32243: 04/10/2018 22:26:41.416 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 616 has been removed (new total is 29)
Line 32253: 04/10/2018 22:26:43.350 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 659 has registered (new total is 30)
Line 32257: 04/10/2018 22:26:43.413 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 656 has registered (new total is 31)
Line 32261: 04/10/2018 22:26:43.732 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 657 has registered (new total is 32)
Line 32268: 04/10/2018 22:26:44.670 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 658 has registered (new total is 33)
Line 32272: 04/10/2018 22:26:44.843 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 653 has registered (new total is 34)
Line 32491: 04/10/2018 22:27:53.985 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 628 has been removed (new total is 33)
Line 32503: 04/10/2018 22:27:56.192 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 663 has registered (new total is 34)
Line 32507: 04/10/2018 22:27:56.397 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 661 has registered (new total is 35)
Line 32514: 04/10/2018 22:27:57.248 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 664 has registered (new total is 36)
Line 32518: 04/10/2018 22:27:57.316 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 660 has registered (new total is 37)
Line 32536: 04/10/2018 22:27:58.780 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 630 has been removed (new total is 36)
Line 32540: 04/10/2018 22:27:58.790 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 662 has registered (new total is 37)
Line 32550: 04/10/2018 22:27:59.287 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 627 has been removed (new total is 36)
Line 32566: 04/10/2018 22:28:01.272 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 667 has registered (new total is 37)
Line 32573: 04/10/2018 22:28:01.559 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 666 has registered (new total is 38)
Line 32583: 04/10/2018 22:28:01.818 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 631 has been removed (new total is 37)
Line 32587: 04/10/2018 22:28:01.912 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 668 has registered (new total is 38)
Line 32597: 04/10/2018 22:28:02.072 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 632 has been removed (new total is 37)
Line 32610: 04/10/2018 22:28:03.098 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 629 has been removed (new total is 36)
Line 32620: 04/10/2018 22:28:04.929 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 669 has registered (new total is 37)
Line 32635: 04/10/2018 22:28:06.100 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 626 has been removed (new total is 36)
Line 32645: 04/10/2018 22:28:08.402 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 673 has registered (new total is 37)
Line 32652: 04/10/2018 22:28:08.511 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 670 has registered (new total is 38)
Line 32656: 04/10/2018 22:28:08.580 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 672 has registered (new total is 39)
Line 32677: 04/10/2018 22:28:10.103 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 635 has been removed (new total is 38)
Line 32687: 04/10/2018 22:28:10.151 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 636 has been removed (new total is 37)
Line 32697: 04/10/2018 22:28:10.163 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 634 has been removed (new total is 36)
Line 32712: 04/10/2018 22:28:10.812 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 637 has been removed (new total is 35)
Line 32716: 04/10/2018 22:28:11.439 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 665 has registered (new total is 36)
Line 32729: 04/10/2018 22:28:12.132 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 633 has been removed (new total is 35)
Line 32749: 04/10/2018 22:28:13.820 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 641 has been removed (new total is 34)
Line 32753: 04/10/2018 22:28:13.867 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 671 has registered (new total is 35)
Line 32757: 04/10/2018 22:28:14.390 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 675 has registered (new total is 36)
Line 32766: 04/10/2018 22:28:14.425 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 639 has been removed (new total is 35)
Line 32774: 04/10/2018 22:28:14.783 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 676 has registered (new total is 36)
Line 32792: 04/10/2018 22:28:18.614 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 677 has registered (new total is 37)
Line 32796: 04/10/2018 22:28:18.784 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 674 has registered (new total is 38)
Line 32805: 04/10/2018 22:28:18.817 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 643 has been removed (new total is 37)
Line 32823: 04/10/2018 22:28:20.244 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 644 has been removed (new total is 36)
Line 32833: 04/10/2018 22:28:20.410 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 638 has been removed (new total is 35)
Line 32840: 04/10/2018 22:28:20.686 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 679 has registered (new total is 36)
Line 32852: 04/10/2018 22:28:22.715 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 682 has registered (new total is 37)
Line 32861: 04/10/2018 22:28:23.693 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 678 has registered (new total is 38)
Line 32871: 04/10/2018 22:28:23.819 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 645 has been removed (new total is 37)
Line 32881: 04/10/2018 22:28:23.842 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 646 has been removed (new total is 36)
Line 32888: 04/10/2018 22:28:24.685 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 681 has registered (new total is 37)
Line 32905: 04/10/2018 22:28:26.400 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 648 has been removed (new total is 36)
Line 32912: 04/10/2018 22:28:26.537 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 683 has registered (new total is 37)
Line 32916: 04/10/2018 22:28:26.799 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 684 has registered (new total is 38)
Line 32929: 04/10/2018 22:28:27.602 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 649 has been removed (new total is 37)
Line 32942: 04/10/2018 22:28:28.945 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 686 has registered (new total is 38)
Line 32946: 04/10/2018 22:28:29.184 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 685 has registered (new total is 39)
Line 32956: 04/10/2018 22:28:29.255 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 647 has been removed (new total is 38)
Line 32966: 04/10/2018 22:28:29.378 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 640 has been removed (new total is 37)
Line 32976: 04/10/2018 22:28:29.407 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 642 has been removed (new total is 36)
Line 32991: 04/10/2018 22:28:31.845 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 687 has registered (new total is 37)
Line 33001: 04/10/2018 22:28:31.966 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 650 has been removed (new total is 36)
Line 33005: 04/10/2018 22:28:32.270 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 680 has registered (new total is 37)
Line 33009: 04/10/2018 22:28:32.437 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 688 has registered (new total is 38)
Line 33029: 04/10/2018 22:28:34.211 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 651 has been removed (new total is 37)
Line 33033: 04/10/2018 22:28:34.353 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 689 has registered (new total is 38)
Line 33049: 04/10/2018 22:28:35.535 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 654 has been removed (new total is 37)
Line 33053: 04/10/2018 22:28:35.846 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 692 has registered (new total is 38)
Line 33060: 04/10/2018 22:28:36.803 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 693 has registered (new total is 39)
Line 33077: 04/10/2018 22:28:38.320 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 652 has been removed (new total is 38)
Line 33090: 04/10/2018 22:28:38.905 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 655 has been removed (new total is 37)
Line 33097: 04/10/2018 22:28:39.513 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 690 has registered (new total is 38)
Line 33101: 04/10/2018 22:28:40.113 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 691 has registered (new total is 39)
Line 33108: 04/10/2018 22:28:41.003 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 694 has registered (new total is 40)
Line 33115: 04/10/2018 22:28:42.069 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 695 has registered (new total is 41)
Line 33122: 04/10/2018 22:28:42.973 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 696 has registered (new total is 42)
Line 33141: 04/10/2018 22:28:44.032 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 659 has been removed (new total is 41)
Line 33154: 04/10/2018 22:28:44.694 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 657 has been removed (new total is 40)
Line 33164: 04/10/2018 22:28:44.701 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 656 has been removed (new total is 39)
Line 33181: 04/10/2018 22:28:45.736 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 658 has been removed (new total is 38)
Line 33191: 04/10/2018 22:28:46.170 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 653 has been removed (new total is 37)
Line 33195: 04/10/2018 22:28:46.184 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 700 has registered (new total is 38)
Line 33202: 04/10/2018 22:28:46.900 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 699 has registered (new total is 39)
Line 33205: 04/10/2018 22:28:46.953 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 697 has registered (new total is 40)
Line 33216: 04/10/2018 22:28:49.196 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 698 has registered (new total is 41)
Line 33233: 04/10/2018 22:28:52.615 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 701 has registered (new total is 42)
Line 46792: 04/10/2018 22:29:57.211 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 661 has been removed (new total is 41)
Line 46804: 04/10/2018 22:29:57.377 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 663 has been removed (new total is 40)
Line 46819: 04/10/2018 22:29:58.239 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 660 has been removed (new total is 39)
Line 46832: 04/10/2018 22:29:58.533 INFO [SparkListenerBus] ExecutorAllocationManager: Existing executor 664 has been removed (new total is 38)
Line 46838: 04/10/2018 22:29:58.952 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 704 has registered (new total is 39)
Line 46842: 04/10/2018 22:29:59.247 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 702 has registered (new total is 40)
Line 46845: 04/10/2018 22:29:59.284 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 703 has registered (new total is 41)
Line 46853: 04/10/2018 22:29:59.450 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 706 has registered (new total is 42)
Line 46860: 04/10/2018 22:30:00.783 INFO [SparkListenerBus] ExecutorAllocationManager: New executor 705 has registered (new total is 43)
Line 46879: