Alerting

Alert fired but I don't know why

riotto
Path Finder

I had an alert that fired which shows a condition that the indexer hadn't received a specific kind of event within the last 5 minutes, but
it had received it. I looked at the _indextime of these events and it shows that they were indeed indexed within those 5 minutes. Is there a log that I can look at that might show if the indexer was doing some kind of housekeeping and the events weren't technically
indexed yet?

Tags (1)
0 Karma

riotto
Path Finder

yes...I posted that I looked at _indextime

0 Karma

gcusello
Esteemed Legend

Hi riotto,
did you verified when you received these events using _indextime?
you can run a search like this

index=your_index
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S"), alert_time=strftime(now(),"%Y-%m-%d %H:%M:%S")
| table _time indextime alert_time

maybe you received them after the alert running.

Bye.
Giuseppe

0 Karma

gcusello
Esteemed Legend

yes but event timestamp (not _indextime) is in the time range of your search?
there are two choices:

  • events arrived after the alert time period (you can exclude this situation comparing indextime with the alert_time);
  • events arrived before the alert period but with a timestamp outside of the above time period (you can exclude this situation comparing _time with the alert_time)

Bye.
Giuseppe

0 Karma

riotto
Path Finder

The alert triggered at 21:08. This alert runs every 2 minutes and looks at the last 5 minutes of indexed events. It counts how many of these specific events were indexed in that last 5 minutes. it counted 0 events. BUT, if I look at that time period 21:03 - 21:08 of when the alert counted 0 events and examine the _time and _indextime of those events, it shows 100s of these events with an _indextime of milliseconds of the _time fpr each of them. Am I looking at this wrong?

0 Karma
Get Updates on the Splunk Community!

Admin Your Splunk Cloud, Your Way

Join us to maximize different techniques to best tune Splunk Cloud. In this Tech Enablement, you will get ...

Cloud Platform | Discontinuing support for TLS version 1.0 and 1.1

Overview Transport Layer Security (TLS) is a security communications protocol that lets two computers, ...

New Customer Testimonials

Enterprises of all sizes and across different industries are accelerating cloud adoption by migrating ...