Splunk Enterprise

Error when pushing bundle to shcluster. Error = "No target specified"

mhofmeester
New Member

We are using a stand-alone deployer to deploy apps to a cluster of 5 search heads. Currently, when trying to push a shcluster bundle from the deployer, we get the error "no target specified".

We're using the following command to push:

./splunk apply shcluster-bundle --answer-yes -target https://[sh-ip]:8089 -preserve-lookups true

We've also tried sequentially:
./splunk apply shcluster-bundle -action stage #no error
./splunk apply shcluster-bundle -action send #same "no target specified" error

Furthermore:
- We've tried to push to all 5 searchheads using ip and hostname alternatively. All resulting in the same error.
- The shcluster has all machines online with an active captain.
- We can also telnet to 8089 from the deployer to the SH's, on both sides the port is active under splunkd.
- We cannot find any log messages on the deployer that indicate what the problem is, even when running with 'splunk start --debug'.
- All folders in ./shcluster/apps/* are owned by splunk.
- We've refreshed the SHClustering pass4SymKey.

Does anyone have an idea, as to why this error is popping up? It's not possible to deploy changes to our cluster at the moment.

Misc info:
- Machines are running RHEL 6.9
- Splunk is running 6.5.5.

0 Karma

tokio13
Path Finder

How did you managed to solve?

0 Karma

jcorcoran508
Path Finder

After more research I use this string, in this case, I would run this string
on SHC1 or SHC3 search heads. Not on the SHC2 server or the local server name you on.


/opt/splunk/bin/splunk apply --answer-yes shcluster-bundle -target https://SHC2:8089 -auth admin:HolySmoke

0 Karma

isoutamo
SplunkTrust
SplunkTrust

You must run this on deployer not on any member of SHC!

Please edit your previous message and remove -auth..... for security reason and after that change you admin password on that environment 😞

On SHC1-SHC3 you should run 

splunk show shcluster-status [--verbose]
splunk show kvstore-status [--verbose]

To see which one is captain and that shc is ok 

btw. You don't need that -auth after you have first time authenticated towards node (e.g. just run this "splunk show shcluster-status) and it will ask user + password from you. Then no needs to add it later on. In security reason I never use that part on command line or if use then remove history as soon command has return. Otherwise that user + password can found from history!

r. Ismo

isoutamo
SplunkTrust
SplunkTrust

Hi

Usually this means that you have some typos in command line (e.g. https -> http, or missing some part of target's hostname etc.).

Personally I prefer to apply bundle towards SH captain. So check first (from any shc node/MC) which one is captain and that SHC and kvstore are ok. Then just apply your bundle from Deployer.

splunk apply shcluster-bundle -target https://<shc captain>:<mgmt_port> [--answer-yes]

r. Ismo 

tokio13
Path Finder

Thanks Ismo for replying!

While a minor adjustment to the command mentioned by you worked and allowed me to move forward, I encountered an authentication error (splunk apply shcluster-bundle --answer-yes -target https://shc-sh1:port)

Which I assume it has to do with what you were talking previously with the other user commenting on this topic.

Error while deploying apps to first member, aborting apps deployment to all members: Error while fetching apps baseline on target=https://shc-sh1:port: Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call no t properly authenticated"}]}

0 Karma

isoutamo
SplunkTrust
SplunkTrust

What are outputs of 

 

 

splunk show shcluster-status
splunk show kvstore-status 

 

 

does those both shows that your SHC is fully functional? 
You should run those on any SHC member node.

0 Karma

maciep
Champion

dumb question, does it matter if you throw the --answer-yes at the end? I mean "no target specified" kinda makes it sound like it's not even parsing the command correctly. just a thought....

0 Karma

mhofmeester
New Member

I've tried running the command without '--answer-yes' to no avail. I can even see the bundle being created in ./splunk/var/run/splunk/deploy/*. It really stops when trying to find a target searchhead to push to.

0 Karma

sbbadri
Motivator

@mhofmeester

1) Check all the search head are up and running.
2) check captain is elected properly by issuing below command on one of the search head
splunk show shcluster-status
3) If all the search head are up and running. Do a rolling restart search head cluster by issuing below command,
splunk rolling-restart shcluster-member from search head captain.

0 Karma

mhofmeester
New Member

We've already checked 1), 2) and 3). However, just to be sure I've followed the steps again:
1) & 2): Using './splunk show shcluster-status' we can see all 5 SH's are up and a dynamic captain has been elected for quite some time.
3): Rolling restart had no effect.

Other interesting fact, other SH clusters in the same network (with their own deployer) are not affected by this issue. The clusters seem to have the same consistent config.

0 Karma

jcorcoran508
Path Finder

I am interested in your solution - can you share ?

0 Karma

SinghK
Builder

You need to add a cluster member name with port 8089 in there 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...