Splunk Search

searching events for "was down for" and averaging the time

jyates76
Explorer

I have events like the below that are saying when a particular pool member was out of rotation for a particular period of time.  What would be an ideal search would be to match all events that have the "was down for" and then the length of time and simply average that, and take the 95th percentile of that duration.   Probably more difficult than it seems and I'm not sure how to approach it.

<133>Feb 13 13:01:33 slot2/US66666-CORE-LTM1.company.COM notice mcpd[8701]: 01070727:5: Pool /Common/pool-generic member /Common/servernamew006:8080 monitor status up. [ /Common/mon-xxx-prod-xxx-liveness: up ] [ was down for 0hr:0min:15sec ]
Labels (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jyates76,

you have to extract the down duratio and then run a simple search:

index=your_index "was down for"
| rex "was\s+down\s+for\s+(?<hours>\d+)hr:(?<minutes>\d+)min:(?<seconds>\d+)sec"
| eval duration=hours*3600+minutes*60+seconds
| timechart perc90(duration) BY host

You can test the regex at https://regex101.com/r/75pRcf/1

then you can use other functions or aggregations.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

See just what you’ve been missing | Observability tracks at Splunk University

Looking to sharpen your observability skills so you can better understand how to collect and analyze data from ...

Weezer at .conf25? Say it ain’t so!

Hello Splunkers, The countdown to .conf25 is on-and we've just turned up the volume! We're thrilled to ...

How SC4S Makes Suricata Logs Ingestion Simple

Network security monitoring has become increasingly critical for organizations of all sizes. Splunk has ...