When using deployer to push configuration updates to cluster members, where do the updated config files go?

[transtrophe]

When using deployer to push configuration updates to cluster members, do the updated config files go into $SPLUNK_HOME/etc/shcluster/users or do I need to specify a user sub-directory under /users? The initial build of the deployer does not have any user sub-dirs under $SPLUNK_HOME/etc/shcluster/users.

[theunf]

I´m having other error :

Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://peer:replication_port: Network-layer error: error:140770FC:SSL routines:SSL23_GET_SERVER_HELLO:unknown protocol

Any clue on that ?

[transtrophe · Accepted solution]

Went back one more time to changing the symmetrical secret key on the deployer and the 3 shc members. Not sure why this didn't work before as mentioned but bottom line and good news is it works now. Obviously a case of UFE - first letter of acronym = user, last letter = error, you fill in the rest - lol.

Thanks esix_splunk and Steve for taking the time to help me on this. Sorry to all that viewed if my obvious fat fingering had you tracking on this thread needlessly.

[Steve_G_]

Why do you want to put outputs.conf under $SPLUNK_HOME/etc/shcluster/users rather than under $SPLUNK_HOME/etc/shcluster/apps? It seems that you would need to put outputs.conf in an apps subdirectory to make it available to the system.

If you do have some need to make a conf file specific to just a single user, then you need to put the file into a user subdirectory under /users, not directly under /users. So, ../users/user1/outputs.conf, not . /users/outputs.conf.

[transtrophe]

Also, I am using port 8089 for the connectivity between the deployed and the shc members. I set up a watch of metatarsal on the shc member I use as the target and executed a netcat from the deployed to this shc member over port 8089, observing the connection establishment in the shc's netstat.

Essentially, verified that the deployed can establish a session over port 8089 with the designated shc target.

Just can't figure out why the shcluster-bundle isn't propagating.

[fabiocaldas]

transtrophe did yout got any solution?

[transtrophe]

I got the issues I brought up in this thread resolved but that resolution has morphed into another set of "show stopper" issues: replication bucket job failures. I am going to open that into its own thread, though, for orderly community board posting etiquette.

[Steve_G_]

I think you might have to contact Support to get to the bottom of this issue. I suspect something went awry when you initially tried to deploy the file from the /users directory, and merely removing it from there somehow wasn't sufficient.

[transtrophe]

I removed the outputs.conf from the $SPLUNK_HOME/etc/shcluster/users dir and tried to execute splunk apply shcluster-bundle with target pointing to o one of the shc members and got this error:

Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://ip-172-31-17-5:8089: Network-layer error: Network connection reset.

I then tried resetting the sh cluster secret key on all 3 of the shc members and the deployer thinking that somehow there was a key mismatch, and got this error:

Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y
Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://ip-172-31-17-5:8089: Network-layer error: Connection refused

[transtrophe]

I'm still a newbie 're: Splunk but, in the timeless words of the Black Knight in Monty Python's Holy Grail, "it's just a flesh wound - it's getting better!"

[transtrophe]

Thanks Steve. I'll try your suggestion and let you know how it goes. I did not remove the outputs.conf from the users dir so maybe that was what failed the push of the sos app.

[transtrophe]

Didn't and still don't have any reason at all to put the outputs.conf under $SPLUNK_HOME/etc/shcluster/users. The documentation in the Best Practices - Forward Search Head Internal Data to the Search Peer (Index) Layer was pretty unclear on this topic, but does lead the reader to the conclusion that configuration files should go under the /users branch versus /apps branch.

I tried the deployment of an app (s.o.s app) loaded in the /apps directory (sos sub-directory, actually), but still get the authentication failure error mentioned earlier.

I haven't tried using different credentials, as esixp suggested earlier because frankly I don't know what other credentials to use beside admin:newadminpass with newadminpass = new password I set on the deployed instance Splunk Web after first logging in and changing "changeme" to my "newadminpass".

[Steve_G_]

Did you delete the configurations from their locations under $SPLUNK_HOME/etc/shcluster/users? I think you will need to do that first, or at least delete the one that's directly under ../users.

I would suggest creating a new subdirectory under ../apps (say "StandaloneConfigs") and putting outputs.conf , and any other standalone conf files, there, rather than adding them to an existing app.

We'll look into clarifying the topic you reference, although on brief perusal I don't see any reference to the user directory in that topic.

[transtrophe]

When I execute splunk commands requiring authentication, I have typically used admin:adminpass executing the command from the splunk account (starting as admin after connecting over SSH, the sudo su to root, then su to splunk).

This is how I did the setup of my index and search head clusters, and what I did with the config-bundle push I am having trouble with here. Should I be specifying a different set of credentials? When I state that I am using admin:adminpass, adminpass is what I specified for the Splunk Web password.

[transtrophe]

I tried putting an updated outputs.conf into the deployer $SPLUNK_HOME/etc/shcluster/users and also into $SPLUNK_HOME/etc/shcluster/users/splunk-system-user (which I made on the deployer with ownership set to splunk:splunk - note, I run all my deployment under the splunk account), and in both cases I get the following error:

Warning: Depending on the configuration changes being pushed, this command might initiate a rolling restart of the cluster members. Please refer to the documentation for the details. Do you wish to continue? [y/n]: y

Error while deploying apps to first member: ConfDeploymentException: Error while fetching apps baseline on target=https://ip-172-31-17-5:8089: Non-200/201 status_code=401; {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

[esix_splunk]

All apps for SHC should be located on the deployer, in the $splunk_home/etc/shcluster folder.

You're getting a 401 error, which means you are not authenticating properly. Check and make sure you're using the correct SHC password in your deployer command.