Scheduled Search - Incorrect Results Depending on When Ran

Path Finder

I currently have a scheduled search which is used to determine the start time and end time of a nightly process. This can be determined by looking in a log for two unique entries; term 1 indicates the process is starting and term 2 indicates it has finished. I have a scheduled search configured for business week to date, set to run nightly an hour or two after the process should be finished:

"term 1" OR "term 2" source="logfile.log" | eval time=strftime(_time,"%r") | timechart span=1d earliest(time) as start_time, latest(time) as end_time | eval Weekday=strftime(_time,"%A %n%m/%d") | table Weekday,start_time,end_time

Anytime I run this search nightly after the process has finished, start_time and end_time are both equal to the start_time. If I run the exact same search the next morning, start_time and end_time are the expected values. Is there something I am missing in the search that would be causing this to happen?

Tags (1)
0 Karma


You say it runs an hour or two after it "should" be finished. Have you verified that you're actually waiting long enough for the process to end and for the unique "term 2" to be logged? (It's also remotely possible that something odd is happening with timezones, but that's just speculation.) Another point--as your search is written, you're not distinguishing between which terms you find. If you happen to find two term 1s in the daily log and no term 2s, then your search will return those times. You should be sure the events are unique and that term 2 will always follow term 1 (and that it completes on the same day), or else you will need to make your search more specific.

If you're sure the event is being logged before the search is run, then it might be that timechart is doing something funky with the time bucketing. I would try this instead:

"term 1" OR "term 2" source="logfile.log" | eval time=strftime(_time,"%r") | eval date=strftime(_time, "%D") | stats earliest(time) as start_time, latest(time) as end_time by date table date,start_time,end_time

0 Karma


We need visibility into logfile.log to assist you. Please paste a snippet of the log file.

0 Karma
Take the 2021 Splunk Career Survey

Help us learn about how Splunk has
impacted your career by taking the 2021 Splunk Career Survey.

Earn $50 in Amazon cash!