Is there a keep alive check for Splunkd in a load balancer environment?

[Ant1D]

Hey,

How can we configure a load balancer to check that splunkd is running on a server before the load balancer forwards data to that server?
I am aware of different types of keepalive probes that load balancers can use (e.g. ICMP, TCP, UDP, HTTP) and it would be good to know how other Splunk users have approached this.

I am aware that Forwarders handle load balancing but Forwarders will not be used in this instance.

Thanks in advance for your help.

[MuS]

Greetings from the future,

there is an app https://splunkbase.splunk.com/app/4395/ now that provides a REST API endpoint that the load balancer can check without authentication and without the need to enable the debug endpoints in web.conf.
It provides also an option to take the Splunk instance out of the load balancer group (the load balancer must support such a thing).

Hope this helps ...

cheers, MuS

[Ant1D]

Thanks for your feedback. I should have specified above that we would like to do this without using Splunk forwarders.

[bmacias84]

I would uses Splunks AutoLoadBalancing on the forwarder with Indexer acknowledgement. This way you minimize data lose. Splunk Forwarders will place none responsive servers into quarantine untill next interval.

[ShaneNewman]

We use a powershell script to run on all of our Windows Splunk servers to do this, although the load balancing we leave up the the universal forwarders and heavy forwarders. The concept is simple enough, we check the status of the service to make sure it is up every 60 seconds. If we find the service is not running, the powershell script starts the service. The results of each run is written to a log we created on each individual server - monitored by UNC by a separate server. Those servers have a separate search running, looking for the absence of data for longer than 3 minutes, signifying the server may be down.

Hope that helps.

[Ant1D]

Thanks for the input