Alerting

Alert fired but I don't know why

riotto
Path Finder

I had an alert that fired which shows a condition that the indexer hadn't received a specific kind of event within the last 5 minutes, but
it had received it. I looked at the _indextime of these events and it shows that they were indeed indexed within those 5 minutes. Is there a log that I can look at that might show if the indexer was doing some kind of housekeeping and the events weren't technically
indexed yet?

Tags (1)
0 Karma

riotto
Path Finder

yes...I posted that I looked at _indextime

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi riotto,
did you verified when you received these events using _indextime?
you can run a search like this

index=your_index
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S"), alert_time=strftime(now(),"%Y-%m-%d %H:%M:%S")
| table _time indextime alert_time

maybe you received them after the alert running.

Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

yes but event timestamp (not _indextime) is in the time range of your search?
there are two choices:

  • events arrived after the alert time period (you can exclude this situation comparing indextime with the alert_time);
  • events arrived before the alert period but with a timestamp outside of the above time period (you can exclude this situation comparing _time with the alert_time)

Bye.
Giuseppe

0 Karma

riotto
Path Finder

The alert triggered at 21:08. This alert runs every 2 minutes and looks at the last 5 minutes of indexed events. It counts how many of these specific events were indexed in that last 5 minutes. it counted 0 events. BUT, if I look at that time period 21:03 - 21:08 of when the alert counted 0 events and examine the _time and _indextime of those events, it shows 100s of these events with an _indextime of milliseconds of the _time fpr each of them. Am I looking at this wrong?

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...