Как адаптировать время ожидания функции init () сценария входа AzureML InferenceConfig? - PullRequest
0 голосов
/ 27 апреля 2020

Я упаковываю модель AzureML, используя метод Model.package. Мой сценарий ввода должен прочитать и обработать некоторые данные в init(), поэтому это может занять некоторое время.

MODEL_NAME = 'abv'
ws = Workspace.from_config()
model = Model(ws, name=MODEL_NAME, version=2)
myenv = Environment.from_conda_specification(name='myenv',
                                             file_path='./env.yaml')
inference_config = InferenceConfig(entry_script='./entry_script.py',
                                   environment=myenv)
package = Model.package(ws, [model],
                        inference_config)
package.wait_for_creation(show_output=True)

После запуска контейнера docker, если появляется следующая ошибка:

...
WORKER TIMEOUT (pid:39)
worker timed out, killing gunicorn
User's init function failed
Encountered Exception Traceback (most recent call last):
  File "/var/azureml-server/aml_blueprint.py", line 162, in register
    main.init()
  File "/var/azureml-app/main.py", line 35, in init
    driver_module.init()
  File "/var/azureml-app/entry_script.py", line 222, in init
    model = MyModel(df, wv)
  File "/var/azureml-app/entry_script.py", line 37, in __init__
    df["TRANSLATED_MANAGEMENT_SUMMARY"], tf_summary)
  File "/var/azureml-app/entry_script.py", line 158, in _restricted_token_embeddings
    stem(token.text).lower() for token in self.nlp(document)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/spacy/language.py", line 439, in __call__
    doc = proc(doc, **component_cfg.get(name, {}))
  File "pipes.pyx", line 396, in spacy.pipeline.pipes.Tagger.__call__
  File "pipes.pyx", line 415, in spacy.pipeline.pipes.Tagger.predict
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
    return self.predict(x)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/feed_forward.py", line 40, in predict
    X = layer(X)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
    return self.predict(x)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/api.py", line 310, in predict
    X = layer(layer.ops.flatten(seqs_in, pad=pad))
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
    return self.predict(x)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/feed_forward.py", line 40, in predict
    X = layer(X)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 167, in __call__
    return self.predict(x)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/model.py", line 131, in predict
    y, _ = self.begin_update(X, drop=None)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/api.py", line 379, in uniqued_fwd
    Y_uniq, bp_Y_uniq = layer.begin_update(X_uniq, drop=drop)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/feed_forward.py", line 46, in begin_update
    X, inc_layer_grad = layer.begin_update(X, drop=drop)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/layernorm.py", line 65, in begin_update
    N, mu, var = _get_moments(self.ops, X)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/thinc/neural/_classes/layernorm.py", line 108, in _get_moments
    var = X.var(axis=1, keepdims=True) + 1e-08
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/numpy/core/_methods.py", line 195, in _var
    x = um.multiply(x, x, out=x)
  File "/azureml-envs/azureml_3078d13611cf6a67d640c421d6d7297f/lib/python3.6/site-packages/gunicorn/workers/base.py", line 195, in handle_abort
    self.cfg.worker_abort(self)
  File "/var/azureml-server/gunicorn_conf.py", line 48, in worker_abort
    sys.exit(3)
SystemExit: 3

Я предполагаю, что это происходит, когда функция init() сценария ввода занимает слишком много времени ( иногда работает нормально, кстати). Как я могу адаптировать тайм-аут?

Добро пожаловать на сайт PullRequest, где вы можете задавать вопросы и получать ответы от других членов сообщества.
...