Why aren't my Splunk Events showing up until ALL time is selected?

[SunilMaharishi]

Hello Team ,

We have one alert scheduled which runs every 5 mins and checks events for last 30 mins to fire. But we are not able to get this working as Events are not visible from the device to Splunk until i select "all time" during search.

What I suspect is — because events are not even visible if there is any time picker — it's not firing an alert when it runs every 5 mins because it is checking with the last 30 minute time range .

In search, if I select "ALL time", I can see the events from the device in Splunk.

I'm unable to find any reason for such behavior. We won't be able to fire an alert if events are not picked up when a specific time picker is selected, and all time is not an option to select to configure the alert as it will search all the events everytime.

Any help would be appreciated

[sduff_splunk]

I suspect your time parsing is incorrect.

Do you have the correct timezones for your data?
Are they log sources, forwarders, indexers and search heads all running NTP and are time syncing correctly?
Does the search work for one user, and if so, what is the timezone that they are using?

When making a scheduled search, it is often good to have some latency in the data you are looking at. Try putting earliest=-5m@m latest=-35m@m to make sure any events that are delayed have 5 minutes to be ingested before they are searched.

[SunilMaharishi]

@sduff . Thanks for response . These logs are coming from firewall and are in GMT ,

we have NTP configured in our enviornment ,

When new events are generated , i can not see those events in splunk when i pick last 5 mins 10mins 1 hour etc , but if i select ALL time i can see those events .

and scheduled search has last 30 mins time configured so that search when runs it doesnt see those events so no email alert fired real time during event generation ,

for that same event i get email alert may be 5 to 7 hours later usually.

[SunilMaharishi]

just to add when these events are visible under normal time picker also that time splunk fire the alert so basically we are getting alert with huge delay in time. i have checked and found that search do complete as scheduled but with zero events as events were not available when last 30 mins time range was selected earlier.