What does this error message mean: "something ... needs splunkd to be up, and splunkd is down"

[lguinn2]

I see this message sometimes, in a variety of forms

This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down.

This command [POST /services/cluster/master/control/control/apply] needs splunkd to be up, and splunkd is down.

This command [GET /services/admin/inputstatus/TailingProcessor:FileStatus] needs splunkd to be up, and splunkd is down

... and many other variations

What does it mean?

[lguinn2 · Accepted solution]

The error message

This command [...] needs splunkd to be up, and splunkd is down.

• could mean that splunkd is down
• could mean that there really is a problem with the tailing processor or whatever component is mentioned

But it almost always means:
"There was a syntax error in your command and the CLI was not smart enough to throw a clear error message. Or maybe splunkd didn't return a specific error code for the CLI to report back to you."

So first i would check
- is there a typo in the command that you typed? This will solve the problem the majority of the time
- is splunkd actually up? This is almost certainly not the problem, but it could happen
- check the splunkd.log and see if there are other error messages that would help you track down the error
- try again to find the typo

Update: I have actually received this error message a couple of times when the command actually worked. My guess is that splunkd returned a warning to the CLI, but the CLI misunderstood and vomited up its standard useless message "...needs splunkd to be up..."

[adane]

Something else that you might try is running your command with -debug, that should reveal a lot more than the logs. That's what helped me discover that I had 

[httpServer]
disableDefaultPort = true

 in ./etc/system/local/server.conf.

[Lowell]

Agreed in general. But I've hit a few scenarios where the error message is bogus (but only for the [GET /services/messages/restart_required/] command), all of the other commands seem to be CLI syntax errors as you point out.

Here's an example:

~/cmaster/bin/splunk edit cluster-config -cluster_label "Lowell Idx Cluster"
The cluster-config property has been edited.
This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down.

I'll also point out that sometimes splunkd.log` show you the exact error, which makes me wonder why the CLI can't just report it? 😉

~/cmaster/bin/splunk edit cluster-config -mode supermaster
This command [POST /services/cluster/config/config] needs splunkd to be up, and splunkd is down.

splunkd.log shows:

12-16-2015 19:20:04.516 +0000 ERROR ClusterStatusHandler - mode=supermaster invalid

[lguinn2]

Thanks @Lowell - I've also gotten this message when the command actually worked.
So. very. frustrating.