Я получаю около 60 Мб данных из входного плагина jdb c (база данных mysql) и пытаюсь загрузить его на мой сервер minio. Я получаю только часть этого.
Для файлов, которые не загружаются, я получаю следующую ошибку:
logstash_1 | [2020-04-23T19:24:28,978][ERROR][logstash.outputs.s3 ][main] Uploading failed, retrying. {:exception=>Seahorse::Client::NetworkingError, :message=>"Net::ReadTimeout", :path=>"/tmp/logstash/dc0fd933-fc1b-43bd-b081-9b1899bc9cd9/events/ls.s3.abd26f2c-a1d6-4588-a387-8e551bfdd9bb.2020-04-23T18.52.part2.txt", :backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:181:in `rbuf_fill'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:157:in `readuntil'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/protocol.rb:167:in `readline'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:40:in `read_status_line'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http/response.rb:29:in `read_new'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/patches.rb:29:in `block in new_transport_request'", "org/jruby/RubyKernel.java:1193:in `catch'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/patches.rb:26:in `new_transport_request'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1474:in `request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/connection_pool.rb:330:in `request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:72:in `block in transmit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:121:in `block in session'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/connection_pool.rb:96:in `session_for'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:119:in `session'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:71:in `transmit'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/net_http/handler.rb:45:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/plugins/content_length.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_request_signer.rb:88:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_request_signer.rb:23:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_host_id.rb:14:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/xml/error_handler.rb:8:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_request_signer.rb:65:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_redirects.rb:15:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:108:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:139:in `retry_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:122:in `retry_if_possible'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/retry_errors.rb:110:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_dualstack.rb:32:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_accelerate.rb:49:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_md5s.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_iad_regional_endpoint.rb:31:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/rest/handler.rb:7:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/user_agent.rb:12:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/plugins/endpoint.rb:41:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/param_validator.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/plugins/raise_response_errors.rb:14:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_dualstack.rb:24:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/s3_accelerate.rb:34:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:20:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/idempotency_token.rb:18:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/aws-sdk-core/plugins/param_converter.rb:20:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/plugins/response_target.rb:21:in `call'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/request.rb:70:in `send_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-core-2.11.451/lib/seahorse/client/base.rb:207:in `block in define_operation_methods'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/file_uploader.rb:42:in `block in put_object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/file_uploader.rb:49:in `open_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/file_uploader.rb:41:in `put_object'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/file_uploader.rb:34:in `upload'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/aws-sdk-resources-2.11.451/lib/aws-sdk-resources/services/s3/object.rb:252:in `upload_file'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.2.0/lib/logstash/outputs/s3/uploader.rb:38:in `upload'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-s3-4.2.0/lib/logstash/outputs/s3/uploader.rb:29:in `block in upload_async'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/java_executor_service.rb:79:in `run'"]}`
Это только если я использую размер файла размером 200 КБ. Если я пытаюсь отправить файл большего размера, он обычно не загружает файл в minio s3.
Если я go в файловой системе контейнера logsta sh docker, я вижу файлы, которые не были отправлены в папку / tmp / logsta sh.
Версия: logsta sh -output-s3 4.2.0 и 4.3.1, logsta sh 7.6.2 Операционная система: ubuntu 18 / docker
Файл конфигурации:
s3{
endpoint => "${S3_HOST}"
access_key_id => "${S3_ACCESS_KEY}"
secret_access_key => "${S3_SECRET_KEY}"
bucket => "twitter-monitoring"
size_file => 204800
time_file => 1
codec => csv{
columns => [ "uid", "id", "accessNumber", "authorHandle", "language", "excerpt", "location", "normalizedLocation"]
charset => "UTF-8"
}
validate_credentials_on_root_bucket => false
prefix => "events"
additional_settings => {
force_path_style => true
follow_redirects => false
}
}
Также подал вопрос здесь: https://github.com/logstash-plugins/logstash-output-s3/issues/222