Why aren't my Splunk Events showing up until ALL time is selected?

Path Finder

Hello Team ,

We have one alert scheduled which runs every 5 mins and checks events for last 30 mins to fire. But we are not able to get this working as Events are not visible from the device to Splunk until i select "all time" during search.

What I suspect is — because events are not even visible if there is any time picker — it's not firing an alert when it runs every 5 mins because it is checking with the last 30 minute time range .

In search, if I select "ALL time", I can see the events from the device in Splunk.

I'm unable to find any reason for such behavior. We won't be able to fire an alert if events are not picked up when a specific time picker is selected, and all time is not an option to select to configure the alert as it will search all the events everytime.

Any help would be appreciated

Tags (1)
0 Karma

Splunk Employee
Splunk Employee

I suspect your time parsing is incorrect.

Do you have the correct timezones for your data?
Are they log sources, forwarders, indexers and search heads all running NTP and are time syncing correctly?
Does the search work for one user, and if so, what is the timezone that they are using?

When making a scheduled search, it is often good to have some latency in the data you are looking at. Try putting earliest=-5m@m latest=-35m@m to make sure any events that are delayed have 5 minutes to be ingested before they are searched.

0 Karma

Path Finder

@sduff . Thanks for response . These logs are coming from firewall and are in GMT ,

we have NTP configured in our enviornment ,

When new events are generated , i can not see those events in splunk when i pick last 5 mins 10mins 1 hour etc , but if i select ALL time i can see those events .

and scheduled search has last 30 mins time configured so that search when runs it doesnt see those events so no email alert fired real time during event generation ,

for that same event i get email alert may be 5 to 7 hours later usually.

0 Karma

Path Finder

just to add when these events are visible under normal time picker also that time splunk fire the alert so basically we are getting alert with huge delay in time. i have checked and found that search do complete as scheduled but with zero events as events were not available when last 30 mins time range was selected earlier.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...