Thanks for both your inputs. The issue has resolved. What it looks like is that I needed to pass in the credentials for the account that I SSHd into the splunk instance, which was admin AND once on the instance to su to the account that launches splunk which is "splunk"; so the splunk account executes the $SPLUNK_HOME/bin/splunk init shcluster-config command with the -auth parameter set to admin:adminspassword.
Anyway, its all up and running now so thanks again for both your inputs.
Also, tried executing the command from the admin account, and got this:
admin@:/opt/splunk/bin$ ./splunk init shcluster-config -replication_port 9997 -mgmt_uri https://:8090 -secret Error setting the real and effective group id:Operation not permitted(1) configured_asPath=splunk configured_asUID=1001 rv__drop_priv_perm=-1 Failed to set effective and real user to value of env var SPLUNK_OS_USER, "splunk"; exiting.: Operation not permitted admin@:/opt/splunk/bin$
It needs to run from the account that splunk runs under (operation not permitted errors).
I think you are confusing internal splunk accounts vs OS based accounts. Do you have a "splunk" OS account or is it "admin"? That command should be run from the OS based account but credentials provided should be the Splunk account (your actually verifying that you are the splunk admin to perform that splunk command).
Within the account that you are going to run splunk from make sure that all settings are correct for that user name. ie. permissions on the entire directory structure under which splunk is installed ie. /opt/splunk. It also need to be set correctly. Also look in /opt/splunk/etc/splunk-launch.conf do you have SPLUNKOSUSER set?
Also make sure you havn't accidently started splunk under the root account. If you have you'll have to chown all the files back to the proper account. On a previously sucessfully running splunk install you shouldn't see any of these errors. It feels like you have a broken installation.
When initializing the cluster, there are two passwords required:
1) User with admin rights
2) Password used for the SHC
Make sure you are using the correct combination of both.
I tried these
root@:/opt/splunk/bin# ./splunk init shcluster-config -replication_port 9997 -mgmt_uri https://:8090 -secret Splunk username: admin Password: Can't create directory "/root/.splunk": Permission denied root@:/opt/splunk/bin# ./splunk init shcluster-config -replication_port 9997 -mgmt_uri https://:8090 -secret Splunk username: admin Password: Login failed root@:/opt/splunk/bin#
Thought the first attempt would work, but it threw that "Can't create directory
"/root/.splunk": Permission denied error.