Splunk Search

How to search the daily average of the top 95% of events and the percentage change?

Path Finder

Hello!

I have two separate searches that I would like to combine into one, someone able to assist, please?

I am trying to accomplish the following: display the 95% top of events daily avg(duration) as well as the percent change of this average between today and yesterday.

This is what I have so far:
For 95% avg: not sure
For %change:

search... | bucket _time span=1d | stats avg(duration) as duration_daily by _time |delta duration_daily as change |eval change_percent=change/(duration_daily-change)*100 |timechart span=1d  first(duration_daily) AS "daily avg", first(change_percent) AS "Change (%)"

Anyway to combine them since I need to see the %change between the average daily values of the 95% of events?

Thank you!

0 Karma

Legend

Try this

base search earliest=-1d@d | eval when=if(_time>relative_time(now(), "@d"), "Today", "Yesterday") | eval Time=strftime(relative_time(now(), "@d"), "%m/%d/%Y") | chart  avg(duration) as  duration_daily over Time by when | eval "Change (%)"=round(Yesterday/Today*100, 2) | fields - Today - Yesterday | appendcols [ search base search earliest=@d | eval Time=strftime(relative_time(now(), "@d"), "%m/%d/%Y") | chart perc5(bytes) as 95b over Time]

Path Finder

Something does not look right on my end. How would the search look like just for the average of events, the top 95?

0 Karma

Legend

This will only show 3 cols Time, "Change (%)" AND 95b (this is the field with 95th Percentile value. To see avg(event), remove the fields - Today - Yesterday The last segment should be

chart perc5(duration) as 95b over Time
0 Karma

Path Finder

Thank you for your input!

0 Karma

SplunkTrust
SplunkTrust

You will need to include a subsearch to accomplish this, but you gotta be careful as the performance takes a hit when doing subsearches. You will pipe the first search into | appendcols [search SEARCH2]

Search 1 = index=search1 * | top(duration)
Search 2 = index=search2. | bucket _time span=1d | stats avg(duration) as duration_daily by _time |delta duration_daily as change |eval change_percent=change/(duration_daily-change)*100 |timechart span=1d first(duration_daily) AS "daily avg", first(change_percent) AS "Change (%)"

Would look like this

index=search1 * | top(duration) | appendcols [search index=search2. | bucket _time span=1d | stats avg(duration) as duration_daily by _time |delta duration_daily as change |eval change_percent=change/(duration_daily-change)*100 |timechart span=1d first(duration_daily) AS "daily avg", first(change_percent) AS "Change (%)"]

Path Finder

Ok, I see. But issue still remains to filter out the bottom 5% and average only the top 95%.

0 Karma

SplunkTrust
SplunkTrust

| stats perc95(your_field)

0 Karma

Path Finder

perc95(duration) will give all the points in the top 95%, correct? Then how can I take the average of those?

0 Karma

SplunkTrust
SplunkTrust

You can take the output of | stats perc95(duration) and pipe it into another command to find the average

| stats perc95(duration) | appendcols [search stats avg(duration)]

0 Karma

Path Finder

Thank you for your input!

0 Karma

SplunkTrust
SplunkTrust

Please accept the answer and/or upvote if this helped you

0 Karma