Splunk Search

How to plot P95,P99,P75, mean and median response times values using tstat?

dsenapaty
Explorer

Hello All,

I need help trying to generate the P95,P99,P75, mean and median response times for the below data using tstats command. Need help with the splunk query. I am dealing with a large data and also building a visual dashboard to my management. So trying to use tstats as searches are faster. Stuck with unable to find these calculations with value of Total_TT in my tstat command. Can someone help me with the query.

 

Sample Data:

2022-09-11 22:00:59,998 INFO -(Success:true)-(Validation:true)-(GUID:68D74EBE-CE3B-7508-6028-CBE1DFA90F8A)-(REQ_RCVD:2022-09-11T22:00:59.051)-(RES_SENT:2022-09-11T22:00:59.989)-(SIZE:2 KB)-(RespSent_TT:0ms)-(Actual_TT:938ms)-(DB_TT:9ms)-(Total_TT:947ms)-(AppServer_TT:937ms)

Labels (1)
0 Karma

jdunlea
Contributor

The data as you have provided it will unfortunately not play well with PREFIX because after the numbers, there is "ms" which converts the value to a string, so we cannot do "p99"/avg, etc using PREFIX.

 

What you would have to do is pull out the Total_TT values as strings, using PREFIX, and then use rtrim() to remove the "ms" at the end, and then multiply the uique values of tota_tt by the number of times they come up, before doing the average/p99, etc. Something like the following should work:

| tstats count where index=YOURINDEX sourcetype=YOURSOURCETYPE by PREFIX(total_tt:) 
| rename "total_tt:" as total_tt 
| eval total_tt=rtrim(total_tt,"ms") 
| eval total_tt_by_count=total_tt*count 
| stats avg(total_tt_by_count) as avg_total_tt p99(total_tt_by_count) as p99_total_tt median(total_tt_by_count) as median_total_tt

 

Let me know if that helps!

Get Updates on the Splunk Community!

Observability Unlocked: Kubernetes & Cloud Monitoring with Splunk IM

Ready to master Kubernetes and cloud monitoring like the pros? Join Splunk’s Growth Engineering team on ...

Index This | What did the zero say to the eight?

June 2025 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this month’s ...

Splunk Observability Cloud's AI Assistant in Action Series: Onboarding New Hires & ...

This is the fifth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...