Splunk Search

Service Downtime Duration

tmarlette
Motivator

I am attempting to find the duration of each downtime instance that has occurred in the last 24 hours, and I am attempting to use the transaction command to do so. I am currently using WMI to query service state, and I'm looking to visualize when the 'State' field changes from "Running" to "Down" and then the duration between the first "Down" State message, and the next "Running" state message.

I'm looking for the results to be in a table that looks kind of like this:
DT1 (time of first down message),DT2 (time of next "Running" message), ,host

Sourcetype=<mysourcetype> Name=<servicename> | transaction State maxpause=10 | timechart max(duration) by Name,host

I don't know if this is the best way to go about this, because my query doesn't seem to be returning the data i'm looking for.

Any help would be greatly appreciated!

0 Karma
1 Solution

linu1988
Champion

Why don't we keep it simple?

Sourcetype=<mysourcetype> | transaction Name startswith="State=Stopped" endswith="State=Running" | stats sum(duration) as "Total Downtime in Seconds" by Name,host

Thanks

View solution in original post

0 Karma

linu1988
Champion

Why don't we keep it simple?

Sourcetype=<mysourcetype> | transaction Name startswith="State=Stopped" endswith="State=Running" | stats sum(duration) as "Total Downtime in Seconds" by Name,host

Thanks

0 Karma

tmarlette
Motivator

This works beautifully. I am just piping this into an eval statement to get the % of downtime per month now. thank you sir!

0 Karma

somesoni2
Revered Legend

Try something like this

sourcetype="servicestatus" | streamstats  window=1 current=f last(State) as Prev  by Name| where NOT State=Prev | transaction Name startswith=State=Down endswith=State=Running | eval UpTime =_time+duration |convert ctime(_time) as DownTime ctime(UpTime) as UpTime | table DownTime, UpTime, Name, host
0 Karma

lguinn2
Legend

Can you give more detail about how you know whether the service is "up" or "down"?

0 Karma

lguinn2
Legend

What about this

sourcetype=mysourcetype 
| eval state = if(State=="Down",0,1)
| sort _time
| xyseries _time servicename state

Then look it as a visualization. This will do all of the servicenames at once, but you could select just a few in the search command.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

[Puzzles] Solve, Learn, Repeat: Character substitutions with Regular Expressions

This challenge was first posted on Slack #puzzles channelFor BORE at .conf23, we had a puzzle question which ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

[Puzzles] Solve, Learn, Repeat: Matching cron expressions

This puzzle (first published here) is based on matching timestamps to cron expressions.All the timestamps ...