Issue with a Search using Modified Time

[TheLighterHalf]

Hey everyone,

I've got a query here that I'm using to find values over 3 different periods of time. Today, yesterday and two days ago. I've made this query into a report and attached it to a dashboard. I've also scheduled that report to be run every hour so that it doesn't create too much load on the server. When I make a change to the query, save it and run it, it works fine and goes to the latest point in time nearest to now. When I wait a while and refresh, that point of time does not change. So for example, I changed this query today at 12:15, saved it and when it ran it gave me data for today up to 12:00. It also showed the data from all of yesterday and all of two days ago like it was supposed to. I waited for about 2 hours and refreshed without making changes to the query and it still only showed me data up to 12:00 even though i have it to be run on a schedule every hour. I've even tried using cron to run the report every 10 minutes to see if it made a difference, Anyone know what might be happening here?

"Order Metrics" earliest=-0d@d latest=now | eval ReportKey="today" | append [search "Order Metrics" earliest=-1d@d latest=-0d@d | eval ReportKey="yesterday" | eval _time=_time+60*60*24]| append [search "Order Metrics" earliest=-2d@d latest=-1d@d | eval ReportKey="two days ago" | eval _time=_time+60*60*24*2] | timechart span=30m sum(order.membership_quantity) by ReportKey

[TheLighterHalf]

Also, for anyone looking at this in the future for reference, here's my search query. I plugged that into search and then changed the time picker to custom > advanced and set earliest to -2d@ and latest to +1@d

"Order Metrics" | timechart sum(order.membership_quantity) span=30m | timewrap 1day

[jplumsdaine22 · Accepted solution]

First up are you attempting to overlay three time series? Have a look at the | timewrap command instead: https://docs.splunk.com/Documentation/Splunk/7.1.0/SearchReference/Timewrap

In terms of your scheduling difficulties, if you run the report, but also run the query ad hoc in a different panel, does it still show data only up to 12pm ?

[TheLighterHalf]

So timewrap worked like a charm! Thanks for that, made the query a lot more manageable. Now I did try a test using two different panels, one a report and one an ad hoc search, and it looks like the report snapped to a particular time and would not update but the ad hoc search did update to the latest time. So its like our reports are not running even though I have it cron scheduled for */15 * * * * *. Any idea what might be happening there? This does seem to be the case for all reports we use on the dashboard, they run once upon creation and then never update with new data even though they are all scheduled.

[jplumsdaine22]

What does the savedsearch.conf entry look like?

[TheLighterHalf]

[Memberships over Time Optimized]
action.email.useNSSubject = 1
alert.track = 0
cron_schedule = */15 * * * *
dispatch.earliest_time = -2d@d
dispatch.latest_time = +1d@d
display.general.type = visualizations
display.page.search.tab = visualizations
display.statistics.show = 0
display.visualizations.charting.chart = line
enableSched = 1
request.ui_dispatch_app = search
request.ui_dispatch_view = search
schedule_priority = higher
search = "Order Metrics" | timechart sum(order.membership_quantity) span=30m | timewrap 1day

[jplumsdaine22]

Two things there.

cron_schedule should look like this:

*/15 * * * * 

If you look in your scheduler logs you will probably see a complaint about "invalid cron" or something.

Also your time range is:
dispatch.earliest_time = -2d@d
dispatch.latest_time = +1d@d

@d snaps to the nearest 12AM. So even if you run the search every 15 minutes the search will be looking at the same time range until the day advances

Fix both of those and you should be good to go!

[TheLighterHalf]

Thanks for your help on this! I think the comment I sent before actually took out the asterisks so it came over as /15 instead of asterisk/15 asterisk asterisk asterisk asterisk. So I believe that should be alright. So in the ad hoc search I've run it will still grab the latest run data point of 11:30 AM while the report sticks to 8:30 AM. They both have the -2d@d and +1d@d. So is there a way to have the report grab the latest run like it does for the ad hoc search? Since the report is run every fifteen minutes why isn't it using the latest run results of (for this example it's 11:53 AM right now) 11:45 AM?