Can't create directory "/opt/splunk/.splunk": No such file or directory

[selina]

I'm currently trying to create a search head cluster for two search head servers while configuring the deployer server.

[Environment Description]
On Search Head Server 1 (10.10.10.5), there are two Splunk daemons installed as follows:

1) Search Head (SH)
   Path: /opt/splunk_sh
    // I'm going to designate this daemon as a deployer member.
2) Indexer Cluster Master (CM)
   Path: /opt/splunk_cm

At this point, the account running each daemon on Search Head Server 1 is 'splunk', which is the same.


On Search Head Server 2 (10.10.10.6), there is one Splunk daemon installed:

1) Search Head (SH)
   Path: /opt/splunk_sh
   // I intend to set this daemon as both a deployer member and a search head captain.


Deployer Server (10.10.10.9)
1) Search Head Deploy
   Path: /opt/splunk


So, with two search head servers and a deployer server in place, when I tried to configure the member settings on Search Head Server 1, I encountered the following error after entering the command:

[Command]

/opt/splunk_sh/bin/splunk init shcluster-config -auth <admin:adminpw> -mgmt_uri https://10.10.10.5:8089 -replication_port 8080 -replication_factor 2 -conf_deploy_fetch_url https://10.10.10.9:8089 -secret <<pw>>

[Command Result]
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details. Can't create directory "/opt/splunk/.splunk": No such file or directory

Please ignore the WARNING as I haven't properly configured the SSL certificate files yet. The problem below is that I'm having difficulty setting the splunk_home path correctly, as indicated by the question title.

While searching through community posts, I tried the following but it didn't work out: Attempt 1) Setting /opt/splunk_sh/etc/splunk-launch.conf I've already set SPLUNK_HOME=/opt/splunk_sh in this conf file when installing the two daemons.

Now, I'm not sure what to do next. Please help me out.

[richgalloway]

If you're new to Splunk, I strongly recommend you NOT install multiple instances of Splunk on the same server.  Doing so is a tricky practice that requires more than just separate subdirectories.  You must also ensure each instance uses different ports and that those ports are configured on other instances correctly.

If you install instance per server then you can keep the home directory as the default /opt/splunk and avoid the problem you are having.

I suspect the problem stems from the home directory assigned to the user running Splunk.  Try changing that directory to /opt/splunk_sh.

[isoutamo]

Hi

It's just like @richgalloway never install more than one splunk installation per server. One instance can have some different roles inside it, but also that is restricted which one can coexistence in one node!

Another issue is, that when you are installing SHC, then minimum node amount is three what it must contains. That restrictions comes from RAFT protocol which manages consistency for SHC.

Only time when you can install several splunk instances on one node is your own personal lab environment. But don't do this in production or even in your official company test environment!

r. Ismo