Monitoring Splunk

What does this error message mean: "something ... needs splunkd to be up, and splunkd is down"

lguinn2
Legend

I see this message sometimes, in a variety of forms

This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down.

This command [POST /services/cluster/master/control/control/apply] needs splunkd to be up, and splunkd is down.

This command [GET /services/admin/inputstatus/TailingProcessor:FileStatus] needs splunkd to be up, and splunkd is down

... and many other variations

What does it mean?

1 Solution

lguinn2
Legend

The error message

This command [...] needs splunkd to be up, and splunkd is down.
  • could mean that splunkd is down
  • could mean that there really is a problem with the tailing processor or whatever component is mentioned

But it almost always means:
"There was a syntax error in your command and the CLI was not smart enough to throw a clear error message. Or maybe splunkd didn't return a specific error code for the CLI to report back to you."

So first i would check
- is there a typo in the command that you typed? This will solve the problem the majority of the time
- is splunkd actually up? This is almost certainly not the problem, but it could happen
- check the splunkd.log and see if there are other error messages that would help you track down the error
- try again to find the typo

Update: I have actually received this error message a couple of times when the command actually worked. My guess is that splunkd returned a warning to the CLI, but the CLI misunderstood and vomited up its standard useless message "...needs splunkd to be up..."

View solution in original post

lguinn2
Legend

The error message

This command [...] needs splunkd to be up, and splunkd is down.
  • could mean that splunkd is down
  • could mean that there really is a problem with the tailing processor or whatever component is mentioned

But it almost always means:
"There was a syntax error in your command and the CLI was not smart enough to throw a clear error message. Or maybe splunkd didn't return a specific error code for the CLI to report back to you."

So first i would check
- is there a typo in the command that you typed? This will solve the problem the majority of the time
- is splunkd actually up? This is almost certainly not the problem, but it could happen
- check the splunkd.log and see if there are other error messages that would help you track down the error
- try again to find the typo

Update: I have actually received this error message a couple of times when the command actually worked. My guess is that splunkd returned a warning to the CLI, but the CLI misunderstood and vomited up its standard useless message "...needs splunkd to be up..."

adane
Engager

Something else that you might try is running your command with -debug, that should reveal a lot more than the logs. That's what helped me discover that I had 

[httpServer]
disableDefaultPort = true

 in ./etc/system/local/server.conf.

0 Karma

Lowell
Super Champion

Agreed in general. But I've hit a few scenarios where the error message is bogus (but only for the [GET /services/messages/restart_required/] command), all of the other commands seem to be CLI syntax errors as you point out.

Here's an example:

~/cmaster/bin/splunk edit cluster-config -cluster_label "Lowell Idx Cluster"
The cluster-config property has been edited.
This command [GET /services/messages/restart_required/] needs splunkd to be up, and splunkd is down.

I'll also point out that sometimes splunkd.log` show you the exact error, which makes me wonder why the CLI can't just report it? 😉

~/cmaster/bin/splunk edit cluster-config -mode supermaster
This command [POST /services/cluster/config/config] needs splunkd to be up, and splunkd is down.

splunkd.log shows:

12-16-2015 19:20:04.516 +0000 ERROR ClusterStatusHandler - mode=supermaster invalid

lguinn2
Legend

Thanks @Lowell - I've also gotten this message when the command actually worked.
So. very. frustrating.

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...

Platform Highlights | January 2023 Newsletter

 January 2023Peace on Earth and Peace of Mind With Business ResilienceAll organizations can start the new year ...