Alerting

Alert fired but I don't know why

riotto
Path Finder

I had an alert that fired which shows a condition that the indexer hadn't received a specific kind of event within the last 5 minutes, but
it had received it. I looked at the _indextime of these events and it shows that they were indeed indexed within those 5 minutes. Is there a log that I can look at that might show if the indexer was doing some kind of housekeeping and the events weren't technically
indexed yet?

Tags (1)
0 Karma

riotto
Path Finder

yes...I posted that I looked at _indextime

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi riotto,
did you verified when you received these events using _indextime?
you can run a search like this

index=your_index
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S"), alert_time=strftime(now(),"%Y-%m-%d %H:%M:%S")
| table _time indextime alert_time

maybe you received them after the alert running.

Bye.
Giuseppe

0 Karma

gcusello
SplunkTrust
SplunkTrust

yes but event timestamp (not _indextime) is in the time range of your search?
there are two choices:

  • events arrived after the alert time period (you can exclude this situation comparing indextime with the alert_time);
  • events arrived before the alert period but with a timestamp outside of the above time period (you can exclude this situation comparing _time with the alert_time)

Bye.
Giuseppe

0 Karma

riotto
Path Finder

The alert triggered at 21:08. This alert runs every 2 minutes and looks at the last 5 minutes of indexed events. It counts how many of these specific events were indexed in that last 5 minutes. it counted 0 events. BUT, if I look at that time period 21:03 - 21:08 of when the alert counted 0 events and examine the _time and _indextime of those events, it shows 100s of these events with an _indextime of milliseconds of the _time fpr each of them. Am I looking at this wrong?

0 Karma
Get Updates on the Splunk Community!

Data Management Digest – November 2025

  Welcome to the inaugural edition of Data Management Digest! As your trusted partner in data innovation, the ...

Introducing Value Insights (Beta): Understand the Business Impact your organization ...

Real progress on your strategic priorities starts with knowing the business outcomes your teams are delivering ...

Seamless IT/OT Security: A Hands-On Look at the Cisco Cyber Vision Splunk Add-on

With just a few clicks, you can ingest critical OT asset details, vulnerabilities, baseline deviations, ...