How can I find an event count of event that are less than a p95 duration?

[mark_groenveld]

I need to identify the count of events that have a duration that is less than the p95 value.

Sample search

index=xyz status=complete | stats p95(dur) as p95Dur

What can I add to the end of the search to id the number of events less than the p95Dur value?

[RobertMarks]

I find doing eventstats on raw data to be tremendously slow. I'd probably compute the percentile in a subsearch and pass it through. Then you only do the percentile computation once.


index=xyz status=complete [ search index=xyz status=complete | stats p95(dur) as p95Dur | eval search = "dur>"+p95Dur | table search]

[mark_groenveld]

Thanks Robert.  I would like to clarify the search as I need the events less than the p95 duration.

Shouldn't the eval section be:  | eval search = "dur<"+p95Dur

[richgalloway]

Use eventstats to compute the p95 value without losing the other fields.

index=xyz status=complete 
| eventstats p95(dur) as p95Dur
| where dur < p95Dur

[mark_groenveld]

Thanks for your response Rich.  Using eventstats took too long to complete to the point it wasn't usable.

[bowesmana]

What do you need to retain from those events? eventstats is a slow operation as it will run on the search head, so the amount of information you need should be minimised before using that, so use the fields command to limit only those fields you need beforehand.

If that is still too slow, the subsearch approach may work for you 

[mark_groenveld]

I need the count of events.

[bowesmana]

Have you tried just

index=xyz status=complete 
| fields dur | fields - _*
| eventstats p95(dur) as p95Dur
| where dur < p95Dur
| stats count

so you only have the dur field in the dataset - I believe that will be significantly faster without having to pull all the data to the search head.