У меня есть журналы ошибок приложения, которые присутствуют в упругом поиске. Пример данных PFB.
{"_index":"error-external-2019-08-08","_type":"el-errors","_id":"CawccGwBeFu2TGbmV247","_score":47.534412,"_source":{"client_id":12345,"data":"{\"responseKey\":\"\",\"clientid\":12345,\"send_to_eaf\":1,\"progressKey\":\"\",\"udt_id\":12,\"proIncrCount\":60,\"wc\":7,\"fileCount\":1,\"cwc\":3,\"url\":\"\",\"jobid\":\"1089\",\"datatype\":[\"text\",\"text\",\"text\",\"text\",\"text\",\"text\",\"text\",\"text\"],\"lists\":[],\"staticlist\":0,\"attr\":[\"domid\",\"EMAIL\",\"MOBILE\",\"att1\",\"att2\",\"att3\",\"att4\",\"foreignkey\"],\"foreignkey\":\"EMAIL\",\"statusFlag\":\"\",\"ts\":\"190808125319\"}","timestamp":1565249072538,"exception_message":"","request_id":"c8d18711-6990-4806-b3a3-52b48fa74d0d","processor_name":"PR1_BULKADD_LIST_STM","error_type":"EXTERNAL","custom_error_message":"Database exception occured","error_code":"ERR20008","message":"","redisString":"","startTime":"1565248999065"}}
Я настроил правило, чтобы получать оповещения, когда какое-то поле содержит указанные значения. Правило pfB yml и config yml.
rule.yml
es_host: 192.92.92.01
es_port: 9200
name: Example rule
type: blacklist
index: error-external-*
timestamp_field: timestamp
timestamp_type: unix_ms
compare_key: processor_name
blacklist:
- "PR1_FP_FILEIMPORTER"
- "PR1_BULKADD_LIST_STM"
- "PR1_BULKEDIT_LIST_STM"
alert:
- "email"
email:
- "ashish.gupta@xyz.co.in"
config.yml
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
minutes: 1
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: 192.92.92.01
# The Elasticsearch port
es_port: 9200
# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1
# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test
# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to Elasticsearch
#use_ssl: True
# Verify TLS certificates
#verify_certs: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
#es_send_get_body_as: GET
# Option basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword
# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
#verify_certs: True
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
writeback_alias: elastalert_alerts
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 5
# Custom logging configuration
# If you want to setup your own logging configuration to log into
# files as well or to Logstash and/or modify log levels, use
# the configuration below and adjust to your needs.
# Note: if you run ElastAlert with --verbose/--debug, the log level of
# the "elastalert" logger is changed to INFO, if not already INFO/DEBUG.
#logging:
# version: 1
# incremental: false
# disable_existing_loggers: false
# formatters:
# logline:
# format: '%(asctime)s %(levelname)+8s %(name)+20s %(message)s'
#
# handlers:
# console:
# class: logging.StreamHandler
# formatter: logline
# level: DEBUG
# stream: ext://sys.stderr
#
# file:
# class : logging.FileHandler
# formatter: logline
# level: DEBUG
# filename: elastalert.log
#
# loggers:
# elastalert:
# level: WARN
# handlers: []
# propagate: true
#
# elasticsearch:
# level: WARN
# handlers: []
# propagate: true
#
# elasticsearch.trace:
# level: WARN
# handlers: []
# propagate: true
#
# '': # root logger
# level: WARN
# handlers:
# - console
# - file
# propagate: false
Когда я тестирую это правило с помощью приведенной ниже команды, мое правило неполучить какие-либо результаты или хиты.
elastalert-test-rule --config config.yaml example_rules/frequency.yaml --days 30
Пример вывода за последние 2 дня
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
Didn't get any results.
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
INFO:elastalert:Queried rule Example rule from 2019-10-29 13:06 IST to 2019-10-29 13:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 13:21 IST to 2019-10-29 13:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 13:36 IST to 2019-10-29 13:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 13:51 IST to 2019-10-29 14:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 14:06 IST to 2019-10-29 14:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 14:21 IST to 2019-10-29 14:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 14:36 IST to 2019-10-29 14:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 14:51 IST to 2019-10-29 15:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 15:06 IST to 2019-10-29 15:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 15:21 IST to 2019-10-29 15:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 15:36 IST to 2019-10-29 15:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 15:51 IST to 2019-10-29 16:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 16:06 IST to 2019-10-29 16:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 16:21 IST to 2019-10-29 16:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 16:36 IST to 2019-10-29 16:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 16:51 IST to 2019-10-29 17:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 17:06 IST to 2019-10-29 17:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 17:21 IST to 2019-10-29 17:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 17:36 IST to 2019-10-29 17:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 17:51 IST to 2019-10-29 18:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 18:06 IST to 2019-10-29 18:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 18:21 IST to 2019-10-29 18:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 18:36 IST to 2019-10-29 18:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 18:51 IST to 2019-10-29 19:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 19:06 IST to 2019-10-29 19:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 19:21 IST to 2019-10-29 19:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 19:36 IST to 2019-10-29 19:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 19:51 IST to 2019-10-29 20:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 20:06 IST to 2019-10-29 20:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 20:21 IST to 2019-10-29 20:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 20:36 IST to 2019-10-29 20:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 20:51 IST to 2019-10-29 21:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 21:06 IST to 2019-10-29 21:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 21:21 IST to 2019-10-29 21:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 21:36 IST to 2019-10-29 21:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 21:51 IST to 2019-10-29 22:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 22:06 IST to 2019-10-29 22:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 22:21 IST to 2019-10-29 22:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 22:36 IST to 2019-10-29 22:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 22:51 IST to 2019-10-29 23:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 23:06 IST to 2019-10-29 23:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 23:21 IST to 2019-10-29 23:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 23:36 IST to 2019-10-29 23:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-29 23:51 IST to 2019-10-30 00:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 00:06 IST to 2019-10-30 00:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 00:21 IST to 2019-10-30 00:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 00:36 IST to 2019-10-30 00:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 00:51 IST to 2019-10-30 01:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 01:06 IST to 2019-10-30 01:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 01:21 IST to 2019-10-30 01:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 01:36 IST to 2019-10-30 01:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 01:51 IST to 2019-10-30 02:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 02:06 IST to 2019-10-30 02:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 02:21 IST to 2019-10-30 02:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 02:36 IST to 2019-10-30 02:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 02:51 IST to 2019-10-30 03:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 03:06 IST to 2019-10-30 03:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 03:21 IST to 2019-10-30 03:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 03:36 IST to 2019-10-30 03:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 03:51 IST to 2019-10-30 04:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 04:06 IST to 2019-10-30 04:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 04:21 IST to 2019-10-30 04:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 04:36 IST to 2019-10-30 04:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 04:51 IST to 2019-10-30 05:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 05:06 IST to 2019-10-30 05:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 05:21 IST to 2019-10-30 05:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 05:36 IST to 2019-10-30 05:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 05:51 IST to 2019-10-30 06:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 06:06 IST to 2019-10-30 06:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 06:21 IST to 2019-10-30 06:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 06:36 IST to 2019-10-30 06:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 06:51 IST to 2019-10-30 07:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 07:06 IST to 2019-10-30 07:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 07:21 IST to 2019-10-30 07:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 07:36 IST to 2019-10-30 07:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 07:51 IST to 2019-10-30 08:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 08:06 IST to 2019-10-30 08:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 08:21 IST to 2019-10-30 08:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 08:36 IST to 2019-10-30 08:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 08:51 IST to 2019-10-30 09:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 09:06 IST to 2019-10-30 09:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 09:21 IST to 2019-10-30 09:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 09:36 IST to 2019-10-30 09:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 09:51 IST to 2019-10-30 10:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 10:06 IST to 2019-10-30 10:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 10:21 IST to 2019-10-30 10:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 10:36 IST to 2019-10-30 10:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 10:51 IST to 2019-10-30 11:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 11:06 IST to 2019-10-30 11:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 11:21 IST to 2019-10-30 11:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 11:36 IST to 2019-10-30 11:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 11:51 IST to 2019-10-30 12:06 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 12:06 IST to 2019-10-30 12:21 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 12:21 IST to 2019-10-30 12:36 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 12:36 IST to 2019-10-30 12:51 IST: 0 / 0 hits
INFO:elastalert:Queried rule Example rule from 2019-10-30 12:51 IST to 2019-10-30 13:06 IST: 0 / 0 hits
Would have written the following documents to writeback index (default is elastalert_status):
elastalert_status - {'rule_name': 'Example rule', 'endtime': datetime.datetime(2019, 10, 30, 7, 36, 9, 750369, tzinfo=tzutc()), 'starttime': datetime.datetime(2019, 10, 29, 7, 36, 9, 750369, tzinfo=tzutc()), 'matches': 0, 'hits': 0, '@timestamp': datetime.datetime(2019, 10, 30, 7, 36, 10, 885732, tzinfo=tzutc()), 'time_taken': 1.1149957180023193}
Пожалуйста, дайте мне знать, я что-то пропустил?