topic Re: What does this error message mean: "something ... needs splunkd to be up, and splunkd is down" in Monitoring Splunk

What does this error message mean: "something ... needs splunkd to be up, and splunkd is down"

lguinn2 — Mon, 09 Nov 2015 21:50:39 GMT

I see this message sometimes, in a variety of forms

This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down.

This command [POST /services/cluster/master/control/control/apply] needs splunkd to be up, and splunkd is down.

This command [GET /services/admin/inputstatus/TailingProcessor:FileStatus] needs splunkd to be up, and splunkd is down

... and many other variations

What does it mean?

Re: What does this error message mean: "something ... needs splunkd to be up, and splunkd is down"

lguinn2 — Mon, 09 Nov 2015 21:55:00 GMT

The error message

This command [...] needs splunkd to be up, and splunkd is down.

could mean that splunkd is down
could mean that there really is a problem with the tailing processor or whatever component is mentioned

But it almost always means:
"There was a syntax error in your command and the CLI was not smart enough to throw a clear error message. Or maybe splunkd didn't return a specific error code for the CLI to report back to you."

So first i would check
- is there a typo in the command that you typed? This will solve the problem the majority of the time
- is splunkd actually up? This is almost certainly not the problem, but it could happen
- check the splunkd.log and see if there are other error messages that would help you track down the error
- try again to find the typo

Update: I have actually received this error message a couple of times when the command actually worked. My guess is that splunkd returned a warning to the CLI, but the CLI misunderstood and vomited up its standard useless message "...needs splunkd to be up..."

Re: What does this error message mean: "something ... needs splunkd to be up, and splunkd is down"

Lowell — Wed, 16 Dec 2015 19:17:45 GMT

Agreed in general. But I've hit a few scenarios where the error message is bogus (but only for the [GET /services/messages/restart_required/] command), all of the other commands seem to be CLI syntax errors as you point out.

Here's an example:

~/cmaster/bin/splunk edit cluster-config -cluster_label "Lowell Idx Cluster"
The cluster-config property has been edited.
This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down.

I'll also point out that sometimes splunkd.log` show you the exact error, which makes me wonder why the CLI can't just report it? 😉

~/cmaster/bin/splunk edit cluster-config -mode supermaster
This command [POST /services/cluster/config/config] needs splunkd to be up, and splunkd is down.

splunkd.log shows:

12-16-2015 19:20:04.516 +0000 ERROR ClusterStatusHandler - mode=supermaster invalid

Re: What does this error message mean: "something ... needs splunkd to be up, and splunkd is down"

lguinn2 — Thu, 17 Dec 2015 00:06:25 GMT

Thanks @Lowell - I've also gotten this message when the command actually worked.
So. very. frustrating.

Re: What does this error message mean: "something ... needs splunkd to be up, and splunkd is down"

adane — Wed, 16 Feb 2022 12:55:10 GMT

Something else that you might try is running your command with -debug, that should reveal a lot more than the logs. That's what helped me discover that I had

[httpServer] disableDefaultPort = true

in ./etc/system/local/server.conf.