Splunk Search

## How to search the daily average of the top 95% of events and the percentage change?

Path Finder

Hello!

I have two separate searches that I would like to combine into one, someone able to assist, please?

I am trying to accomplish the following: display the 95% top of events daily avg(duration) as well as the percent change of this average between today and yesterday.

This is what I have so far:
For 95% avg: not sure
For %change:

``````search... | bucket _time span=1d | stats avg(duration) as duration_daily by _time |delta duration_daily as change |eval change_percent=change/(duration_daily-change)*100 |timechart span=1d  first(duration_daily) AS "daily avg", first(change_percent) AS "Change (%)"
``````

Anyway to combine them since I need to see the %change between the average daily values of the 95% of events?

Thank you!

Tags (5)
Legend

Try this

``````base search earliest=-1d@d | eval when=if(_time>relative_time(now(), "@d"), "Today", "Yesterday") | eval Time=strftime(relative_time(now(), "@d"), "%m/%d/%Y") | chart  avg(duration) as  duration_daily over Time by when | eval "Change (%)"=round(Yesterday/Today*100, 2) | fields - Today - Yesterday | appendcols [ search base search earliest=@d | eval Time=strftime(relative_time(now(), "@d"), "%m/%d/%Y") | chart perc5(bytes) as 95b over Time]
``````
Path Finder

Something does not look right on my end. How would the search look like just for the average of events, the top 95?

Legend

This will only show 3 cols Time, "Change (%)" AND 95b (this is the field with 95th Percentile value. To see avg(event), remove the `fields - Today - Yesterday` The last segment should be

``````chart perc5(duration) as 95b over Time
``````
Path Finder

Thank you for your input!

SplunkTrust

You will need to include a subsearch to accomplish this, but you gotta be careful as the performance takes a hit when doing subsearches. You will pipe the first search into `| appendcols [search SEARCH2]`

Search 1 = `index=search1 * | top(duration)`
Search 2 = `index=search2. | bucket _time span=1d | stats avg(duration) as duration_daily by _time |delta duration_daily as change |eval change_percent=change/(duration_daily-change)*100 |timechart span=1d first(duration_daily) AS "daily avg", first(change_percent) AS "Change (%)"`

Would look like this

``````index=search1 * | top(duration) | appendcols [search index=search2. | bucket _time span=1d | stats avg(duration) as duration_daily by _time |delta duration_daily as change |eval change_percent=change/(duration_daily-change)*100 |timechart span=1d first(duration_daily) AS "daily avg", first(change_percent) AS "Change (%)"]
``````
Path Finder

Ok, I see. But issue still remains to filter out the bottom 5% and average only the top 95%.

SplunkTrust

`| stats perc95(your_field)`

Path Finder

perc95(duration) will give all the points in the top 95%, correct? Then how can I take the average of those?

SplunkTrust

You can take the output of `| stats perc95(duration)` and pipe it into another command to find the average

`| stats perc95(duration) | appendcols [search stats avg(duration)]`

Path Finder

Thank you for your input!

SplunkTrust

Please accept the answer and/or upvote if this helped you