Splunk Search
Highlighted

Compare two periods, results shows only from today

Path Finder

I want to search over two periods to measure the average response time of the access data of GET requests. The period is over 15 minutes, broken up in 10s pieces. I only see the data of "now" but just one hit on "yesterday". The sub query results 7237 events. No issue on max-sub-events or timeouts. This is my query.

index=prd_access sourcetype=access:web:agw earliest=03/01/2013:11:00:00 latest=03/01/2013:11:15:00 GET | eval key="now" | append [search index=prd_access sourcetype=access:web:agw earliest=02/28/2013:11:00:00 latest=02/28/2013:11:15:00 GET  | eval _time=_time+60*60*24 | eval key="yesterday" ] | timechart span=10s avg(duration) by key
Tags (2)
Highlighted

Re: Compare two periods, results shows only from today

SplunkTrust
SplunkTrust

A general hint on time computations:

... | eval _time = relative_time(_time, "+1d") | ...
0 Karma
Highlighted

Re: Compare two periods, results shows only from today

SplunkTrust
SplunkTrust

Wow that's creepy. I think it has to be a bug, maybe in append or maybe in something deeper. But whatever it is, it seems like it happens if you alter the _time values while you're still in the subsearch brackets. What happens if you do is that most of the rows get yanked out of existence and never get appended. Why most of them and not all of them, I don't know. I've run lots of different searches with different data and timechart/stats etc... and it really seems to be caused only by monkeying with _time values while in the square brackets of an append.

Here's a more generic example. This search should work, but it doesn't. Most of the "yesterday" events get mysteriously removed.

* earliest="@d+11h" latest="@d+11h+15min" | eval marker="today" | append [search * earliest="-1d@d+11h" latest="-1d@d+11h+15min" | eval marker="yesterday" | eval _time=_time+(60*60*24)] | timechart count by marker

but this very similar search works perfectly.

* earliest="@d+11h" latest="@d+11h+15min" | eval marker="today" | append [search * earliest="-1d@d+11h" latest="-1d@d+11h+15min" | eval marker="yesterday"] | eval _time=if(marker=="yesterday",_time+(60*60*24),_time) | timechart count by marker

And if you go back to that first search and you just remove the eval clause, suddenly you get all the events from yesterday coming out of append correctly.

You can do it with any command, not just with timechart. Here we should end up with two buckets - each with 1000 events (assuming you generally have >1000 events per 15min period in your main index). But this one is missing most of the "yesterday" events.

* earliest="@d+11h" latest="@d+11h+15min" | head 1000 | eval marker="today" | append [search * earliest="-1d@d+11h" latest="-1d@d+11h+15min" | eval marker="yesterday" | eval _time=_time+(60*60*24) | head 1000] | stats count by marker

and this one has them all.

* earliest="@d+11h" latest="@d+11h+15min" | head 1000 | eval marker="today" | append [search * earliest="-1d@d+11h" latest="-1d@d+11h+15min" | eval marker="yesterday" | head 1000] | eval _time=if(marker=="yesterday",_time+(60*60*24),_time) | stats count by marker

View solution in original post