Since upgrading from 5 to 6, one of my dashboards started behaving "strangely", and I have distilled it down to this.
If I have a dashboard that uses "stats" and "first"
<dashboard> <label>TimeTest</label> <description>TimeTest</description> <row> <table> <searchString> index=_internal |stats first(_time) as f, last(_time) as l by sourcetype |eval d = f - l |fieldformat f = strftime(f, "%c") |fieldformat l = strftime(l, "%c") |table sourcetype f l d </searchString> <earliestTime>-2mon</earliestTime> <latestTime>now</latestTime> </table> </row> </dashboard>
This produces some strange results for me when I run it I get cases where "first" is further in the past than "last" - giving negative values for 'd'.
If I then click "Open In Search" - the same results are shown (as expected), but is the then click on the magnifying glass to do the search again.......I get sensible values for all with positive values for 'd'
The above case does take a while to run, but it is example of what I am experiencing in a form that others can, hopefully, reproduce.
All help much appreciated.
and it should give the negative result as per the query as f < l. Moreover when calculating this big range in the query you can bucket the time which will give your reult bit faster as you are not looking for granular detail.
index=_internal|bucket _time span=6h|stats...
Slightly off topic, but I'm curious - how does
bucket speed that up?
To add something to the topic, you can replace your eval with
range(_time) in the
if you bucket, the stats will have very less data to compare, I have done it in some of my graphs where the granular level details doesn't really affect the final outcome very much.
e.g.consider on 17th you may have 1000 events.
if you do a stats then it will go through each timestamp to findout which is the min and max rather the first and last. But if you bucket it to 1 hours and considering 50 events per hour the stats input will become only 20 events which is faster right? this is what i understand and in my dashboard it takes way less when i compare monthly data.
I have worked around the issue using min/max. But that does not deflect from the fact that is is an issue.
The point being I should receive the same output from a dashboard as a search window and I don't.
Unless there is something I am fundamentally doing wrong, then I guess it is a bug
What is the value are you getting? i had the same issue while doing same sort of calculation. Splunk doesn't provide you correct time difference as it always calculate on epoch time format.
I had some help earlier
The issue is not restricted to time, the original issue I had was with page counts on printers - I created the problem above with time just to illustrate the issue.
w.r.t value I was getting, it was negative, and more importantly different to the one I got when I opened up the search in a search bar.
Here's a thought that may be related: I've observed a change in how a 7-day dashboard is filled since upgrading to Splunk 6. In Splunk 5, it would fill sequentially from now back to 7 days ago. In Splunk 6, data appears from the past before that sequential fill arrived there.
That's with a standalone Splunk, so no remote search peer delivering data faster than another peer.
This may or may not affect how first() and last() are calculated.
first() and last() are literally the first and last records processed by your Search Head.
There is no guarantee on order here. It's literally a race condition on what shows up first from the indexers. If you want to guarantee order, you're going to have to apply a sorting criteria, or use some other ordinal function like min() or max().
The great thing about first() and last() are that they are easy to compute and require almost no memory. Once you start introducing a sort, the search is going to take up more resources (something has to keep an ordered list).
The search you want here is actually the following. It is fast and requires no aggregation. It returns the earliest and latest times grouped by sourcetype. Enjoy!
| metadata type=sourcetypes index=_internal