Ошибка сброса соединения после завершения юнит-тестов PySpark - PullRequest
Новая
       12

Ошибка сброса соединения после завершения юнит-тестов PySpark

0 голосов
/ 28 февраля 2020

У меня есть небольшой пакет модульного тестирования для моих заданий PySpark с использованием библиотеки python unittest. Выполнение тестов с python -m unittest успешно запускает тесты, но после завершения тестов PySpark выдает ConnectionResetError. Тесты и все утверждения проходят без проблем. Я делаю что-то не так в своем методе разрыва класса?

Вот код: 


class TestUDF(TestCase):
    @staticmethod
    def create_testing_pyspark_session():
        return SparkSession.builder.master('local').getOrCreate()

    @classmethod
    def setUpClass(self):
        self.spark = TestUDF.create_testing_pyspark_session()

    @classmethod
    def tearDownClass(self):
        self.spark.stop()

Ошибка здесь: 

....
----------------------------------------------------------------------
Ran 5 tests in 221.271s

OK
SUCCESS: The process with PID 6496 (child process of PID 6268) has been terminated.
SUCCESS: The process with PID 6268 (child process of PID 5368) has been terminated.
SUCCESS: The process with PID 5368 (child process of PID 3556) has been terminated.
ERROR:root:Exception while sending command.
Traceback (most recent call last):
  File "C:\Users\gauthama\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\java_gateway.py", line 1152, in send_command
    answer = smart_decode(self.stream.readline()[:-1])
  File "C:\Users\gauthama\AppData\Local\Programs\Python\Python37\lib\socket.py", line 589, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\gauthama\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\java_gateway.py", line 985, in send_command
    response = connection.send_command(command)
  File "C:\Users\gauthama\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\java_gateway.py", line 1164, in send_command
    "Error while receiving", e, proto.ERROR_ON_RECEIVE)
py4j.protocol.Py4JNetworkError: Error while receiving
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:56584)
Traceback (most recent call last):
  File "C:\Users\gauthama\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\gauthama\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\java_gateway.py", line 1067, in start
    self.socket.connect((self.address, self.port))
ConnectionRefusedError: [WinError 10061] No connection could be made because the target machine actively refused it
ERROR:py4j.java_gateway:An error occurred while trying to connect to the Java server (127.0.0.1:56584)
Traceback (most recent call last):
  File "C:\Users\gauthama\AppData\Local\Programs\Python\Python37\lib\site-packages\py4j\java_gateway.py", line 929, in _get_connection
    connection = self.deque.pop()
IndexError: pop from an empty deque
...