Monitoring Splunk

What does this error message mean: "something ... needs splunkd to be up, and splunkd is down"

lguinn2
Legend

I see this message sometimes, in a variety of forms

This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down.

This command [POST /services/cluster/master/control/control/apply] needs splunkd to be up, and splunkd is down.

This command [GET /services/admin/inputstatus/TailingProcessor:FileStatus] needs splunkd to be up, and splunkd is down

... and many other variations

What does it mean?

1 Solution

lguinn2
Legend

The error message

This command [...] needs splunkd to be up, and splunkd is down.
  • could mean that splunkd is down
  • could mean that there really is a problem with the tailing processor or whatever component is mentioned

But it almost always means:
"There was a syntax error in your command and the CLI was not smart enough to throw a clear error message. Or maybe splunkd didn't return a specific error code for the CLI to report back to you."

So first i would check
- is there a typo in the command that you typed? This will solve the problem the majority of the time
- is splunkd actually up? This is almost certainly not the problem, but it could happen
- check the splunkd.log and see if there are other error messages that would help you track down the error
- try again to find the typo

Update: I have actually received this error message a couple of times when the command actually worked. My guess is that splunkd returned a warning to the CLI, but the CLI misunderstood and vomited up its standard useless message "...needs splunkd to be up..."

View solution in original post

lguinn2
Legend

The error message

This command [...] needs splunkd to be up, and splunkd is down.
  • could mean that splunkd is down
  • could mean that there really is a problem with the tailing processor or whatever component is mentioned

But it almost always means:
"There was a syntax error in your command and the CLI was not smart enough to throw a clear error message. Or maybe splunkd didn't return a specific error code for the CLI to report back to you."

So first i would check
- is there a typo in the command that you typed? This will solve the problem the majority of the time
- is splunkd actually up? This is almost certainly not the problem, but it could happen
- check the splunkd.log and see if there are other error messages that would help you track down the error
- try again to find the typo

Update: I have actually received this error message a couple of times when the command actually worked. My guess is that splunkd returned a warning to the CLI, but the CLI misunderstood and vomited up its standard useless message "...needs splunkd to be up..."

View solution in original post

Lowell
Super Champion

Agreed in general. But I've hit a few scenarios where the error message is bogus (but only for the [GET /services/messages/restart_required/] command), all of the other commands seem to be CLI syntax errors as you point out.

Here's an example:

~/cmaster/bin/splunk edit cluster-config -cluster_label "Lowell Idx Cluster"
The cluster-config property has been edited.
This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down.

I'll also point out that sometimes splunkd.log` show you the exact error, which makes me wonder why the CLI can't just report it? 😉

~/cmaster/bin/splunk edit cluster-config -mode supermaster
This command [POST /services/cluster/config/config] needs splunkd to be up, and splunkd is down.

splunkd.log shows:

12-16-2015 19:20:04.516 +0000 ERROR ClusterStatusHandler - mode=supermaster invalid

lguinn2
Legend

Thanks @Lowell - I've also gotten this message when the command actually worked.
So. very. frustrating.

0 Karma
Register for .conf21 Now! Go Vegas or Go Virtual!

How will you .conf21? You decide! Go in-person in Las Vegas, 10/18-10/21, or go online with .conf21 Virtual, 10/19-10/20.