Текущая настройка - конвейер фабрики данных Azure должен запускаться каждые 15 минут. Запустите некоторые записные книжки Databricks на постоянно включенном интерактивном кластере данных.
Проблема, с которой здесь сталкиваются - сбой этого конвейера после 4-5 запусков. Из-за проблем в Spark Driver. Нет операторов Collect, которые могут привести к заполнению памяти драйвера. Журналы ошибок показывают проблему, когда Драйвер пытается записать информацию во внутреннее метасторское хранилище (которое автоматически управляется Databricks). Этот поток вызывает нарушение предельных значений GC и вызывает полный GC. В результате драйвер отключается, и ноутбук запускается с ошибкой.
Вот журналы -
19/11/06 04:56:47 ERROR DatabricksMain$DBUncaughtExceptionHandler: Uncaught exception in thread db-atomic-read-worker-5095!
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.Arrays.copyOf(Arrays.java:3332)
at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:596)
at java.lang.StringBuilder.append(StringBuilder.java:190)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFSpan(ObjectInputStream.java:3506)
at java.io.ObjectInputStream$BlockDataInputStream.readUTFBody(ObjectInputStream.java:3414)
at java.io.ObjectInputStream$BlockDataInputStream.readUTF(ObjectInputStream.java:3226)
at java.io.ObjectInputStream.readString(ObjectInputStream.java:1905)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1564)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at java.util.Hashtable.readObject(Hashtable.java:1213)
at sun.reflect.GeneratedMethodAccessor36.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:1170)
at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2178)
at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2069)
at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1573)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431)
at org.apache.commons.lang3.SerializationUtils.clone(SerializationUtils.java:94)
at org.apache.spark.SparkContext$$anon$2.childValue(SparkContext.scala:370)
at org.apache.spark.SparkContext$$anon$2.childValue(SparkContext.scala:366)
at java.lang.ThreadLocal$ThreadLocalMap.<init>(ThreadLocal.java:391)
at java.lang.ThreadLocal$ThreadLocalMap.<init>(ThreadLocal.java:298)
at java.lang.ThreadLocal.createInheritedMap(ThreadLocal.java:255)
at java.lang.Thread.init(Thread.java:420)
at java.lang.Thread.init(Thread.java:349)
at java.lang.Thread.<init>(Thread.java:511)
at sun.security.ssl.SSLSocketImpl$NotifyHandshakeThread.<init>(SSLSocketImpl.java:2675)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1096)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
19/11/06 04:56:47 ERROR DatabricksMain$DBUncaughtExceptionHandler: OutOfMemoryError in thread db-atomic-read-worker-5095! Killing thread now.
19/11/06 04:56:47 WARN TrapExitSecurityManager: Called "System.exit(15)" in db-atomic-read-worker-5095!
Stack Trace:
java.lang.Thread.getStackTrace(Thread.java:1559)
com.databricks.backend.daemon.driver.TrapExitSecurityManager.checkExit(DriverLocal.scala:686)
java.lang.Runtime.halt(Runtime.java:273)
com.databricks.DatabricksMain$DBUncaughtExceptionHandler.uncaughtException(DatabricksMain.scala:363)
java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1057)
java.lang.ThreadGroup.uncaughtException(ThreadGroup.java:1052)
java.lang.Thread.dispatchUncaughtException(Thread.java:1959)
19/11/06 04:56:47 WARN TrapExitSecurityManager: Allowed to exit because this is OOM!
19/11/06 04:56:52 INFO StaticConf$: DB_HOME: /databricks
19/11/06 04:56:53 INFO DriverDaemon$: ========== driver starting up ==========
19/11/06 04:56:53 INFO DriverDaemon$: Java: Private Build 1.8.0_222
19/11/06 04:56:53 INFO DriverDaemon$: OS: Linux/amd64 4.15.0-1050-azure
19/11/06 04:56:53 INFO DriverDaemon$: CWD: /databricks/driver
Проблемы с подключением к неуправляемому метастазу -
urrent allocation: Map(1414820437514047686 -> 1, 289483405015881873 -> 175)
Ideal allocation: Map(1414820437514047686 -> 88, 289483405015881873 -> 88)
Starved pools: Map(1414820437514047686 -> 98.420017518)
19/11/06 04:55:37 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 588 to 10.139.64.20:49530
19/11/06 04:55:29 ERROR BoneCP: Failed to acquire connection to jdbc:mariadb://consolidated-westeurope-prod-metastore-addl-1.mysql.database.azure.com:3306/organization4787651615040525?trustServerCertificate=true&useSSL=true. Sleeping for 7000 ms. Attempts left: 5
java.sql.SQLNonTransientConnectionException: Could not connect to consolidated-westeurope-prod-metastore-addl-1.mysql.database.azure.com:3306 : Connection reset
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:161)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:106)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1036)
at org.mariadb.jdbc.internal.util.Utils.retrieveProxy(Utils.java:490)
at org.mariadb.jdbc.MariaDbConnection.newConnection(MariaDbConnection.java:144)
at org.mariadb.jdbc.Driver.connect(Driver.java:90)
at java.sql.DriverManager.getConnection(DriverManager.java:664)
at java.sql.DriverManager.getConnection(DriverManager.java:208)
at com.jolbox.bonecp.BoneCP.obtainRawInternalConnection(BoneCP.java:361)
at com.jolbox.bonecp.BoneCP.obtainInternalConnection(BoneCP.java:269)
at com.jolbox.bonecp.ConnectionHandle.<init>(ConnectionHandle.java:242)
at com.jolbox.bonecp.PoolWatchThread.fillConnections(PoolWatchThread.java:115)
at com.jolbox.bonecp.PoolWatchThread.run(PoolWatchThread.java:82)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to consolidated-westeurope-prod-metastore-addl-1.mysql.database.azure.com:3306 : Connection reset
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:161)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.connException(ExceptionMapper.java:79)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:724)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connect(AbstractConnectProtocol.java:402)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1032)
... 13 more
Caused by: java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:975)
at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1761)
at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:124)
at sun.security.ssl.Handshaker.kickstart(Handshaker.java:1079)
at sun.security.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1479)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1346)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.handleConnectionPhases(AbstractConnectProtocol.java:676)
... 15 more
19/11/06 04:55:37 WARN PreemptionMonitor: Preempted 43/43 tasks from 289483405015881873.
19/11/06 04:55:53 WARN PreemptionMonitor: Attempting to preempt 43 tasks from overallocated pools.
19/11/06 04:55:53 INFO PreemptionMonitor: Current allocation state:
Current max parallelism: 176
Буду признателенлюбой ответ на вопрос -
1- Любой параметр кластера заданий зажигания / блоков данных, который я могу настроить, чтобы избежать таких сбоев драйверов?
2-Как избежать накопления памяти этим подключенным демономв Метастор. Очистите память после отправки задания, вызвав какой-то локальный GC.
3- Где я могу увидеть / контролировать этот неуправляемый Metastore?