topic Re: 'splunk status' return code in Splunk Search

'splunk status' return code

ianformanek — Wed, 30 Nov 2011 16:07:41 GMT

When I issue 'splunk status' on Linux, the exit code is 0 even when splunk is not running. This makes it hard to use it in automation - Chef in my case, which relies on the return code to tell it whether the service is up/down. Is there other good way to do this?

Re: 'splunk status' return code

Takajian — Thu, 01 Dec 2011 08:26:20 GMT

I am not sure why 'splunk status' command does not work in your environment. But the other way to monitor splunk process will be "ps" command may be useful.

A normal default Splunk will start up:

・" Two "splunkd" (or "splunkd.exe") processes.
One does indexing, and the other helps launch other processes as necessary
・ " SplunkWeb, which runs inside of "python" (or "pythonservice.exe")

So, you can use following command to monitor those processes.

ps aux | grep splunk | grep -v grep

Re: 'splunk status' return code

ianformanek — Thu, 01 Dec 2011 09:48:10 GMT

Here is the best approach I ended up to simulate returning non-0 exit code when one of the splunk services is not running:

expr `service splunk status | grep 'is not running' | wc -l` == 0

Re: 'splunk status' return code

ianformanek — Thu, 01 Dec 2011 09:49:24 GMT

Thanks Takajian, yes that works. I wanted to avoid doing it this way though, as it is error prone (just naming the server 'my-splunk-server' can easily make the ps result always appear as if splunk is running).

Just to clarify - 'splunk status' does work correctly and reports status for both processes, just not via the exit code. Below is a way I ended up doing it.

Re: 'splunk status' return code

MHibbin — Thu, 01 Dec 2011 17:46:51 GMT

ianformanek,

Building on your answer, would it not be better to a be a little more explicit and use

To check just splunkweb...
... $SPLUNK_HOME/bin/splunk status splunkweb | grep 'is not running' | ...

To check just splunkd (not including helpers)
... $SPLUNK_HOME/bin/splunk status splunkd | grep splunkd| grep 'is not running' | ...

You could also check helpers are running (used for running scripts, etc.)
... $SPLUNK_HOME/bin/splunk status splunkd | grep 'splunk helpers' | grep 'are running'...

Doing this could prove more reliable. As one service could crash without stopping the other (or someone could kill one process but not the other).

Re: 'splunk status' return code

ianformanek — Fri, 02 Dec 2011 07:51:15 GMT

Indeed, depending on the usecase, this may be better. In my case, I want to check if all the services are running to determine if splunk service start needs to be called (that would only start those not running) - hence the check for any ocurrence of 'is not running'.

Re: 'splunk status' return code

tpaulsen — Fri, 08 Jun 2012 08:15:37 GMT

Hello, this seems to be a bug. Splunk exit code is clearly not LSB conform. At least on Linux. We experience the same problem.

http://refspecs.linuxbase.org/LSB_3.1.1/LSB-Core-generic/LSB-Core-generic/iniscrptact.html

Re: 'splunk status' return code

Takol1 — Sat, 07 Jul 2012 03:09:09 GMT

If you use,
splunk status splunkd

The return code is 0 when daemon is running and 3 for not running. But if you use,
splunk status splunkweb

The return code is always 0 no matter web service is running or not. I think it is a bug.

Re: 'splunk status' return code

jrodman — Tue, 09 Feb 2016 01:31:33 GMT

Yes indeed. This is why splunk status changed to returning 3 when the splunkd service is down.

Re: 'splunk status' return code

jrodman — Tue, 09 Feb 2016 01:32:08 GMT

I think this got fixed in the intervening years, but in any event it is no longer relevant, as splunkweb is no longer a separate service.