Alerting

Alert fired but I don't know why

riotto
Path Finder

I had an alert that fired which shows a condition that the indexer hadn't received a specific kind of event within the last 5 minutes, but
it had received it. I looked at the _indextime of these events and it shows that they were indeed indexed within those 5 minutes. Is there a log that I can look at that might show if the indexer was doing some kind of housekeeping and the events weren't technically
indexed yet?

Tags (1)
0 Karma

riotto
Path Finder

yes...I posted that I looked at _indextime

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi riotto,
did you verified when you received these events using _indextime?
you can run a search like this

index=your_index
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S"), alert_time=strftime(now(),"%Y-%m-%d %H:%M:%S")
| table _time indextime alert_time

maybe you received them after the alert running.

Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

yes but event timestamp (not _indextime) is in the time range of your search?
there are two choices:

  • events arrived after the alert time period (you can exclude this situation comparing indextime with the alert_time);
  • events arrived before the alert period but with a timestamp outside of the above time period (you can exclude this situation comparing _time with the alert_time)

Bye.
Giuseppe

0 Karma

riotto
Path Finder

The alert triggered at 21:08. This alert runs every 2 minutes and looks at the last 5 minutes of indexed events. It counts how many of these specific events were indexed in that last 5 minutes. it counted 0 events. BUT, if I look at that time period 21:03 - 21:08 of when the alert counted 0 events and examine the _time and _indextime of those events, it shows 100s of these events with an _indextime of milliseconds of the _time fpr each of them. Am I looking at this wrong?

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

What Is Splunk? Here’s What You Can Do with Splunk

Hey Splunk Community, we know you know Splunk. You likely leverage its unparalleled ability to ingest, index, ...

Level Up Your .conf25: Splunk Arcade Comes to Boston

With .conf25 right around the corner in Boston, there’s a lot to look forward to — inspiring keynotes, ...

Manual Instrumentation with Splunk Observability Cloud: How to Instrument Frontend ...

Although it might seem daunting, as we’ve seen in this series, manual instrumentation can be straightforward ...