Monitoring Splunk

Is there a keep alive check for Splunkd in a load balancer environment?

Ant1D
Motivator

Hey,

How can we configure a load balancer to check that splunkd is running on a server before the load balancer forwards data to that server?
I am aware of different types of keepalive probes that load balancers can use (e.g. ICMP, TCP, UDP, HTTP) and it would be good to know how other Splunk users have approached this.

I am aware that Forwarders handle load balancing but Forwarders will not be used in this instance.

Thanks in advance for your help.

0 Karma

MuS
SplunkTrust
SplunkTrust

Greetings from the future,

there is an app https://splunkbase.splunk.com/app/4395/ now that provides a REST API endpoint that the load balancer can check without authentication and without the need to enable the debug endpoints in web.conf.
It provides also an option to take the Splunk instance out of the load balancer group (the load balancer must support such a thing).

Hope this helps ...

cheers, MuS

0 Karma

Ant1D
Motivator

Thanks for your feedback. I should have specified above that we would like to do this without using Splunk forwarders.

0 Karma

bmacias84
Champion

I would uses Splunks AutoLoadBalancing on the forwarder with Indexer acknowledgement. This way you minimize data lose. Splunk Forwarders will place none responsive servers into quarantine untill next interval.

0 Karma

ShaneNewman
Motivator

We use a powershell script to run on all of our Windows Splunk servers to do this, although the load balancing we leave up the the universal forwarders and heavy forwarders. The concept is simple enough, we check the status of the service to make sure it is up every 60 seconds. If we find the service is not running, the powershell script starts the service. The results of each run is written to a log we created on each individual server - monitored by UNC by a separate server. Those servers have a separate search running, looking for the absence of data for longer than 3 minutes, signifying the server may be down.

Hope that helps.

0 Karma

Ant1D
Motivator

Thanks for the input

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...