searching events for "was down for" and averaging the time

jyates76 — Fri, 16 Feb 2024 12:44:45 GMT

I have events like the below that are saying when a particular pool member was out of rotation for a particular period of time. What would be an ideal search would be to match all events that have the "was down for" and then the length of time and simply average that, and take the 95th percentile of that duration. Probably more difficult than it seems and I'm not sure how to approach it.

<133>Feb 13 13:01:33 slot2/US66666-CORE-LTM1.company.COM notice mcpd[8701]: 01070727:5: Pool /Common/pool-generic member /Common/servernamew006:8080 monitor status up. [ /Common/mon-xxx-prod-xxx-liveness: up ] [ was down for 0hr:0min:15sec ]

host = US66666-core-ltm1.company.com
source = /var/log/HOSTS/US66666-core-ltm1.company.com/xxx.xxx.com-syslog.log
sourcetype = syslog_alb

Re: searching events for "was down for" and averaging the time

gcusello — Fri, 16 Feb 2024 17:16:19 GMT

Hi @jyates76,

you have to extract the down duratio and then run a simple search:

index=your_index "was down for" | rex "was\s+down\s+for\s+(?<hours>\d+)hr:(?<minutes>\d+)min:(?<seconds>\d+)sec" | eval duration=hours*3600+minutes*60+seconds | timechart perc90(duration) BY host

You can test the regex at https://regex101.com/r/75pRcf/1

then you can use other functions or aggregations.

Ciao.

Giuseppe

topic Re: searching events for "was down for" and averaging the time in Splunk Search

searching events for "was down for" and averaging the time

Re: searching events for "was down for" and averaging the time