topic Re: Looking for a REST status OK for load balancer healthcheck in Deployment Architecture

Looking for a REST status OK for load balancer healthcheck

davebo1896 — Wed, 03 Jun 2015 19:54:54 GMT

I'm looking for a basic heartbeat or healthcheck for search head status, in order to put behind a load balancer.
This should not require authentication.
The closest thing I've found is in the REST API /services/server/info , but that provides way more information than is needed.
Isn't there a simple call that will return "OK" ?

Re: Looking for a REST status OK for load balancer healthcheck

MuS — Wed, 03 Jun 2015 21:33:37 GMT

Hi davebo1896,

you can add a bit of SPL to this REST call, like this:

| REST /services/server/info | stats count AS myCount | eval Status=if(myCount=="1", "OK" , "PANIK")| table Status

to get what you want or you just check the TCP port of the search head from load balancer. Because if the Web server from Splunk is down your search head is not usable by any user.

Hope this helps ...

cheers, MuS

Re: Looking for a REST status OK for load balancer healthcheck

davebo1896 — Thu, 04 Jun 2015 13:41:07 GMT

We were hoping for something a bit more sophisticated than checking the TCP port. We want to know that the service is capable of returning something sensible, albeit simple.
For now we're looking at using /debug/echo?ping=OK and parsing for the OK, but that is less than optimal.
We may put an nginx server in front of Splunk Web to return a simple OK (or not), but that adds an extra layer that could cause false readings (both positive and negative).

Re: Looking for a REST status OK for load balancer healthcheck

lpolo — Wed, 17 Aug 2016 16:31:35 GMT

I facing the same situation.
What end point you end up using for health check?

Thanks,
Lp

Re: Looking for a REST status OK for load balancer healthcheck

MuS — Wed, 17 Aug 2016 22:23:59 GMT

@lpolo, look at the comment by davebo1896 you can use

 http[s]://YourSplunkServerName:YourSplunksPortHere/en-GB/debug/echo?ping=OK

and parse the output for the ping OK response.

cheers, MuS

Re: Looking for a REST status OK for load balancer healthcheck

lpolo — Thu, 18 Aug 2016 14:59:24 GMT

It does not work in 6.4.1. The response is:
Unauthorized to access this resource.

Re: Looking for a REST status OK for load balancer healthcheck

lpolo — Thu, 18 Aug 2016 16:08:51 GMT

The approach I took is this one:

nc -zv splunkServer 514
found 0 associations
found 1 connections:
1: flags=82
outif en3
src x.x.x.x port 56747
dst x.x.x.x port 514
rank info not available
TCP aux info available
Connection to 172.24.236.98 port 514 [tcp/shell] succeeded!

You also could do:

https://yourSplunkIndexer:8089/services/server/info/server-info

Thanks,
Lp

Re: Looking for a REST status OK for load balancer healthcheck

MuS — Thu, 18 Aug 2016 20:20:14 GMT

Sure it does 😉 You need to enable the debug endpoints first - they were turned off for security reasons. To enable them change the web.conf

enableWebDebug = true|false
* Controls the visibility of the debug endpoints (i.e., /debug/**splat).
* Defaults to false

Re: Looking for a REST status OK for load balancer healthcheck

MuS — Fri, 01 Mar 2019 02:03:39 GMT

Greetings from the future,

there is an app https://splunkbase.splunk.com/app/4395/ now that provides a REST API endpoint that the load balancer can check without authentication and without the need to enable the debug endpoints in web.conf.
It provides also an option to take the Splunk instance out of the load balancer group (the load balancer must support such a thing).

Hope this helps ...

cheers, MuS