topic service status of a service including Disaster Recovery situation in Splunk Search

service status of a service including Disaster Recovery situation

ashraf_sj — Wed, 24 Apr 2024 14:09:30 GMT

Just in a situation where I have 2 servers, where 1 is active and the other is passive. I had to deploy the TA on both the servers and report the service status of a service.

So the active server would be reporting the service is "Running" and the passive server would say the service is "stopped"

I have tried writing up a SPL but my only worry is if there is a situation when the service stops on the active server how to get it reported. or if there is no data from the active server. There should be atleast 1 server reporting the service is "Running" always. Only during the DR situation the server name would change

index=mday source="service_status.ps1" sourcetype=service_status os_service="App_Service" host=*papp01 | stats values(host) AS active_host BY status | where status=="Running" | append [ search index = mday source =service_status.ps1 sourcetype = service_status os_service="App_Service" host=*papp01 | stats latest(status) AS status by host,os_service,service_name ] | filldown active_host | where active_host=host AND status!="Running" | table host,active_host,os_service,service_name,status

Any help is much appreciated

Re: service status of a service including Disaster Recovery situation

renjith_nair — Thu, 25 Apr 2024 13:59:49 GMT

There are multiple methods to achieve this. However, lets first try it in a simpler way

index=mday source="service_status.ps1" sourcetype=service_status os_service="App_Service" host=*papp01 |stats latest(status) AS status by host |eventstats values(status) as _status |eval OverallStatus=if(mvcount(_status) < 2 OR isnull(mvfind(_status,"Running")),"Down","Good")

Steps

- count the status values

- If the count is less than 2 : meaning only one of the status from Running/Stopped is present

- OR Running status is not available, we are setting the overall status as down.

In this way, we can handle multiple situations where one of the server is down or both are reporting down or even both are reporting Running (active & passive)

Demonstrated with a dummy search

Try changing the status of HostA or HostB and see the results.

Re: service status of a service including Disaster Recovery situation

ashraf_sj — Mon, 29 Apr 2024 08:10:17 GMT

Thanks @renjith_nair , this works, I have used the OverallStatus as condition to alert. Thanks a lot and much appreciated.