Невозможно собрать Spark из источника из-за OOME

Question

Я скачал spark-2.4.0.tgz, и он не может пройти модуль spark-sql сборки.

Выполнены следующие оптимизации в моей среде env сборки:

• Запуск цинка
• Бег с -DskipTests
• Убитые основные программы захвата памяти
  • Top сообщает о 3,8 Г физической памяти доступно во время работы

• Явно выделил процессу много места

  `MAVEN_OPTS="-Xmx3500m" mvn -DskipTests package`
  

• Но Activity Monitor показал, что он использует только 1.5GB: поэтому процесс имелмного доступной неиспользуемой памяти - как с точки зрения system, так и из собственной выделенной памяти

Другие детали:

• на Macos High Sierra
• 16 ГБ физической ОЗУ
• Я обычно запускаю на 10 ГБ больше программ, чем было запущено в то время (только 11 ГБ Memory и Swap используется

Воттрассировка стека:

[warn] /shared/spark/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala:315: method apply in object ProcessingTime is deprecated: use Trigger.ProcessingTime(interval)
[warn]       StartStream(ProcessingTime("10 seconds"), new StreamManualClock),
[warn]                   ^
[warn] /shared/spark/sql/core/src/test/scala/org/apache/spark/sql/streaming/StreamSuite.scala:356: method apply in object ProcessingTime is deprecated: use Trigger.ProcessingTime(interval)
[warn]       StartStream(ProcessingTime("10 seconds"), new StreamManualClock(60 * 1000)),
[warn]                   ^
java.lang.OutOfMemoryError: GC overhead limit exceeded
    at scala.tools.nsc.backend.icode.BasicBlocks$BasicBlock.<init>(BasicBlocks.scala:464)
    at scala.tools.nsc.backend.icode.Members$Code.newBlock(Members.scala:97)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase$Context.newBlock(GenICode.scala:1907)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.genLoadIf(GenICode.scala:354)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genLoad(GenICode.scala:536)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genStat(GenICode.scala:181)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase$$anonfun$genStat$1.apply(GenICode.scala:155)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase$$anonfun$genStat$1.apply(GenICode.scala:155)
    at scala.collection.LinearSeqOptimized$class.foldLeft(LinearSeqOptimized.scala:124)
    at scala.collection.immutable.List.foldLeft(List.scala:84)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.scala$tools$nsc$backend$icode$GenICode$ICodePhase$$genLoad(GenICode.scala:922)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:123)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:71)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:148)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:98)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:71)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:89)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.gen(GenICode.scala:67)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.apply(GenICode.scala:63)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$applyPhase$1.apply$mcV$sp(Global.scala:467)
    at scala.tools.nsc.Global$GlobalPhase.withCurrentUnit(Global.scala:458)
    at scala.tools.nsc.Global$GlobalPhase.applyPhase(Global.scala:467)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:425)
    at scala.tools.nsc.Global$GlobalPhase$$anonfun$run$1.apply(Global.scala:425)
    at scala.collection.Iterator$class.foreach(Iterator.scala:891)
    at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
    at scala.tools.nsc.Global$GlobalPhase.run(Global.scala:425)
    at scala.tools.nsc.backend.icode.GenICode$ICodePhase.run(GenICode.scala:55)
    at scala.tools.nsc.Global$Run.compileUnitsInternal(Global.scala:1528)
    at scala.tools.nsc.Global$Run.compileUnits(Global.scala:1513)
    at scala.tools.nsc.Global$Run.compileSources(Global.scala:1508)
    at scala.tools.nsc.Global$Run.compile(Global.scala:1609)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Spark Project Parent POM 2.4.0:
[INFO]
[INFO] Spark Project Parent POM ........................... SUCCESS [  2.569 s]
[INFO] Spark Project Tags ................................. SUCCESS [  1.136 s]
[INFO] Spark Project Sketch ............................... SUCCESS [  0.499 s]
[INFO] Spark Project Local DB ............................. SUCCESS [  0.658 s]
[INFO] Spark Project Networking ........................... SUCCESS [  1.903 s]
[INFO] Spark Project Shuffle Streaming Service ............ SUCCESS [  0.503 s]
[INFO] Spark Project Unsafe ............................... SUCCESS [  0.590 s]
[INFO] Spark Project Launcher ............................. SUCCESS [  2.484 s]
[INFO] Spark Project Core ................................. SUCCESS [ 16.434 s]
[INFO] Spark Project ML Local Library ..................... SUCCESS [  0.651 s]
[INFO] Spark Project GraphX ............................... SUCCESS [  1.977 s]
[INFO] Spark Project Streaming ............................ SUCCESS [  4.197 s]
[INFO] Spark Project Catalyst ............................. SUCCESS [ 17.791 s]
[INFO] Spark Project SQL .................................. FAILURE [03:49 min]
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 ................... SKIPPED
[INFO] Kafka 0.10+ Source for Structured Streaming ........ SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Integration for Kafka 0.10 Assembly .......... SKIPPED
[INFO] Spark Avro ......................................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  04:41 min
[INFO] Finished at: 2018-12-04T04:37:20-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) on project spark-sql_2.11: Execution scala-test-compile-first of goal net.alchim31.maven:scala-maven-plugin:3.2.2:testCompile failed.: CompileFailed -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-sql_2.11