Tensorflow / Tensorboard на s3 - PullRequest
       19

Tensorflow / Tensorboard на s3

0 голосов
/ 31 января 2020

Мы записываем журналы тензорной доски нашего тренировочного процесса на s3, и пока я хотел бы взглянуть на это. Поэтому я выполняю следующую команду:

tensorboard --logdir=s3://mlflow/1/7327710663534420b09581b9fa601bfc/artifacts/tensorboard_logs/ --bind_all

Я знаю, что этот каталог существует, потому что я проверил с помощью инструмента s3cmd.

Мои переменные env, которые я экспортировал ранее:

AWS_ACCESS_KEY_ID="<MyKey>"
AWS_SECRET_ACCESS_KEY="<MySecret>"
S3_ENDPOINT="<MyCustomEndpoint>"
S3_USE_HTTPS="1"
S3_VERIFY_SSL="0"

И некоторые журналы:

2020-01-31 16:43:54.140206: I tensorflow/core/platform/s3/aws_logging.cc:54] Initializing config loader against fileName /Users/taawech0//.aws/config and using profilePrefix = 1
2020-01-31 16:43:54.140242: I tensorflow/core/platform/s3/aws_logging.cc:54] Initializing config loader against fileName /Users/taawech0//.aws/credentials and using profilePrefix = 0
2020-01-31 16:43:54.140250: I tensorflow/core/platform/s3/aws_logging.cc:54] Setting provider to read credentials from /Users/taawech0//.aws/credentials for credentials file and /Users/taawech0//.aws/config for the config file , for use with profile default
2020-01-31 16:43:54.140278: I tensorflow/core/platform/s3/aws_logging.cc:54] Creating AWSHttpResourceClient with max connections2 and scheme http
2020-01-31 16:43:54.140289: I tensorflow/core/platform/s3/aws_logging.cc:54] Initializing CurlHandleContainer with size 2
2020-01-31 16:43:54.140299: I tensorflow/core/platform/s3/aws_logging.cc:54] Creating Instance with default EC2MetadataClient and refresh rate 300000
2020-01-31 16:43:54.140303: I tensorflow/core/platform/s3/aws_logging.cc:54] Added EC2 metadata service credentials provider to the provider chain.
2020-01-31 16:43:54.140316: I tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2020-01-31 16:43:54.140364: I tensorflow/core/platform/s3/aws_logging.cc:54] Initializing CurlHandleContainer with size 25
2020-01-31 16:43:54.140418: I tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2020-01-31 16:43:54.140528: I tensorflow/core/platform/s3/aws_logging.cc:54] Pool grown by 2
2020-01-31 16:43:54.140534: I tensorflow/core/platform/s3/aws_logging.cc:54] Connection has been released. Continuing.
2020-01-31 16:43:54.270240: E tensorflow/core/platform/s3/aws_logging.cc:60] HTTP response code: 404
Exception name: 
Error message: No response body.
5 response headers:
cache-control : must-revalidate,no-cache,no-store
content-type : text/html;charset=iso-8859-1
date : Fri, 31 Jan 2020 15:43:57 GMT
x-amz-id-2 : 
x-amz-request-id : 8abebd2a:16ff02dd20c:3755:3dc
2020-01-31 16:43:54.270288: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2020-01-31 16:43:54.270354: I tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2020-01-31 16:43:54.270446: I tensorflow/core/platform/s3/aws_logging.cc:54] Connection has been released. Continuing.
TensorBoard 2.1.0 at http://UM01549:6006/ (Press CTRL+C to quit)
2020-01-31 16:43:54.406342: I tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2020-01-31 16:43:54.406515: I tensorflow/core/platform/s3/aws_logging.cc:54] Connection has been released. Continuing.
2020-01-31 16:43:54.448773: E tensorflow/core/platform/s3/aws_logging.cc:60] HTTP response code: 404
Exception name: 
Error message: No response body.
5 response headers:
cache-control : must-revalidate,no-cache,no-store
content-type : text/html;charset=iso-8859-1
date : Fri, 31 Jan 2020 15:43:58 GMT
x-amz-id-2 : 
x-amz-request-id : 8abebd2d:16ff0369a72:371b:722
2020-01-31 16:43:54.448829: W tensorflow/core/platform/s3/aws_logging.cc:57] If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.

В этот момент я как будто очень запутался. Спасибо за вашу помощь.

...