URLError массовой загрузки данных в App Engine: «Сброс соединения по пиру»

Question

Я пытаюсь загрузить некоторые записи в локальное хранилище данных, используя appcfg.py

Фактически вставлено только небольшое количество записей, и я получаю следующий вывод (с кучей ошибок):

       $ appcfg.py upload_data --config_file=bulkloader.yaml --filename=/output.csv --kind=AutoCompleteIndex --url=http://localhost:8084/remote_api .
        Application: theapp1; version: dev.
        Uploading data records.
        [INFO    ] Logging to bulkloader-log-20100808.170223
        [INFO    ] Throttling transfers:
        [INFO    ] Bandwidth: 250000 bytes/second
        [INFO    ] HTTP connections: 8/second
        [INFO    ] Entities inserted/fetched/modified: 20/second
        [INFO    ] Batch Size: 10
        [INFO    ] Opening database: bulkloader-progress-20100808.170223.sql3
        Please enter login credentials for localhost
        Email: kjuyg
        Password for kjuyg: 
        [INFO    ] Connecting to localhost:8084/remote_api
        [INFO    ] Starting import; maximum 10 entities per post
        .......................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .............[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        ................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put
            'datastore_v3', 'Put', put_request, put_response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 155, in MakeSyncCall
            self._MakeRealSyncCall(service, call, request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 167, in _MakeRealSyncCall
            encoded_response = self._server.Send(self._path, encoded_request)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/appengine_rpc.py", line 346, in Send
            f = self.opener.open(req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 391, in open
            response = self._open(req, data)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 409, in _open
            '_open', req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 369, in _call_chain
            result = func(*args)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1173, in http_open
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................[ERROR   ] Retrying on non-fatal URL error: [Errno 54] Connection reset by peer
        Traceback (most recent call last):
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 695, in PerformWork
            transfer_time = self._TransferItem(thread_pool)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 852, in _TransferItem
            self.request_manager.PostEntities(self.content)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/bulkloader.py", line 1296, in PostEntities
            datastore.Put(entities)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 289, in Put
            'datastore_v3', 'Put', req, datastore_pb.PutResponse(), rpc)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/datastore.py", line 193, in _MakeSyncCall
            rpc.check_success()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_stub_map.py", line 474, in check_success
            self.__rpc.CheckSuccess()
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/api/apiproxy_rpc.py", line 149, in _WaitImpl
            self.request, self.response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 223, in MakeSyncCall
            handler(request, response)
          File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/ext/remote_api/remote_api_stub.py", line 349, in _Dynamic_Put

.... etc .....
            return self.do_open(httplib.HTTPConnection, req)
          File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1148, in do_open
            raise URLError(err)
        URLError: <urlopen error [Errno 54] Connection reset by peer>
        .......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................^C..[INFO    ] An error occurred. Shutting down...
        ..........
        [INFO    ] 19580 entites total, 0 previously transferred
        [INFO    ] 18570 entities (3593152 bytes) transferred in 225.1 seconds
        [INFO    ] Some entities not successfully transferred

Есть идеи, в чем может быть проблема?

Answer 1

У меня была точно такая же проблема на моем Mac. Я решил это, создав прокси-сервер, добавив в файл httpd-vhost.conf следующее (в etc / apache2 / extra /):

<Location /remote_api>
  ProxyPass http://localhost:8888/remote_api
</Location>

при условии, что ваш тестовый сервер работает на порту 8888.

Тогда я бегу:

sudo apachectl restart

для перезапуска сервера apache.

Теперь эта команда («Съемка» является одним из элементов данных):

/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/appcfg.py upload_data /
--kind=Survey --file=Survey.csv /
--url=http://localhost/remote_api --email=test@example.com /
--config_file=config.yml 

дает этот вывод:

[INFO    ] Logging to bulkloader-log-20121114.100837
[INFO    ] Throttling transfers:
[INFO    ] Bandwidth: 250000 bytes/second
[INFO    ] HTTP connections: 8/second
[INFO    ] Entities inserted/fetched/modified: 20/second
[INFO    ] Batch Size: 10
Password for test@example.com: 
[INFO    ] Opening database: bulkloader-progress-20121114.100837.sql3
[INFO    ] Connecting to localhost/remote_api
[INFO    ] Starting import; maximum 10 entities per post
....
[INFO    ] 36 entities total, 0 previously transferred
[INFO    ] 36 entities (15402 bytes) transferred in 1.7 seconds
[INFO    ] All entities successfully transferred

Как и должно быть.

Answer 2

Посмотрите логи вашего сервера. Вы получаете превышение размера кучи или исключение OutOfMemmory? Это произойдет на локальном сервере dev, если ваша загрузка велика, потому что сервер dev хранит хранилище данных в основной памяти. Сервер будет зависать немного, а затем вы получите сброс соединения. У моего сервера разработки не хватило памяти с всего лишь 10 МБ данных хранилища данных, так как он использует много дополнительной оперативной памяти для организации и хранения метаданных. Вы можете увеличить размер кучи, используя ключ -Xmx.