'urllib' на бесконечном цикле

Question

Некоторое время я пытался использовать cloud9, чтобы дать мне дополнительную руку на повторяющихся задачах на моей работе. Тем не менее, большинство из этих задач связаны с очисткой веб-страниц, и мне трудно использовать библиотеку urllib.

Это не работает даже для простых кодов. Это продолжается вечно, и я не могу найти причины для этого. Буду признателен за некоторые советы ...

from urllib.request import urlopen
import json
import ssl

# Ignore SSL certificate errors
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

url = ('Enter - ')
html = urlopen(url, context=ctx).read()
str_data = open(html).read()
json_data = json.loads(str_data)

for entry in json_data:

name = entry[0];
    title = entry[1];
    print((name, title))

Answer 1

Сделайте себе одолжение и замените его на Запросы . Ваш пример упрощается до:

import requests

url = "..."
json_data = requests.get(url).json()

for entry in json_data:
    name = entry[0]
    title = entry[1]
    print(name, title)

Это также убирает из вашего кода огромную вероятность ошибок.

Answer 2

Новый код

import requests
import json

url = "-"
json_data = requests.get(url).json()
info = json.loads(json_data)

print('User count:', len(info))
print(info)

Режим отладки:

[IKP3db-g] 15:43:52,971733 - INFO - IKP3db 1.4.1 - Inouk Python Debugger for CPython 3.6+
[IKP3db-g] 15:43:52,972714 - INFO - IKP3db listening on 127.0.0.1:15471
[IKP3db-g] 15:43:53,003016 - INFO - Connected with 127.0.0.1:50936
Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
TimeoutError: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 376, in _make_request
    self._validate_conn(conn)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 994, in _validate_conn
    conn.connect()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connection.py", line 334, in connect
    conn = self._new_conn()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.VerifiedHTTPSConnection object at 0x7fa2bb9729e8>: Failed to establish a new connection: [Errno 110] Connection timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='ot.cloud.mapsfinancial.com', port=443): Max retries exceeded with url: /pegasus/api/consulta/caixa/saldos/carteira/DVG1%20FIA/2018-01-08 (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa2bb9729e8>: Failed to establish a new connection: [Errno 110] Connection timed out',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ikp3db.py", line 2047, in main
  File "/usr/local/lib/python3.6/dist-packages/ikp3db.py", line 1526, in _runscript
    exec(statement, globals, locals)
  File "<string>", line 1, in <module>
  File "/home/ubuntu/environment/Testing.py", line 5, in <module>
    json_data = requests.get(url).json()
  File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/api.py", line 75, in get
    return request('get', url, params=params, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/api.py", line 60, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/ubuntu/.local/lib/python3.6/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='ot.cloud.mapsfinancial.com', port=443): Max retries exceeded with url: xxxxxxxx (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7fa2bb9729e8>: Failed to establish a new connection: [Errno 110] Connection timed out',))[IKP3db-g] 15:46:04,668520 - INFO - Uncaught exception. Entering post mortem debugging

Answer 3

Не предоставляя много отладочной информации, можно только предположить, что вы, вероятно, истекло время ожидания, поскольку вы не предоставляете функции urlopen время ожидания.

Чтобы избежать этого, установите разумное время ожидания для вашего Функция urlopen, чтобы он не работал бесконечно.