Could not create Splunk settings directory at '/root/.splunk'

[splunklearner]

Anytime I try to do anything with my deployment server I get this error:

An error occurred: Could not create Splunk settings directory at '/root/.splunk'

This includes the command -  ./splunk reload deploy-server. 

We have AWS EC2 instances hosted for all components and opening it via SSM and login via sudo -i. Tried to give sudo chown -R splunk:splunk /opt/splunk/bin.. still the same issue.

• And one more doubt - if we edit in etc/deployment-apps reload is enough right to distribute the updated configurations to manager? But when I restart configurations are reflecting in manager not sure why reload is throwing this error?

[kiran_panchavat]

sudo su splunk
./splunk reload deploy-server

[kiran_panchavat]

Yes, you can update the configurations in `/opt/splunk/etc/deployment-apps` and push them to the clients. That should be sufficient. 

[kiran_panchavat]

@splunklearner 

It's basically: Splunk is not running as root (which is good), but you're logged in as root (which is not so good).
When you try to login (which is required to reload the deployment server), Splunk saves your login (which would be a cookie if you we're using a browser) in a folder in your home directory (/root/.splunk/). But as Splunk does not run as root, it has no permissions to do that, therefore the error message.

Solution: Do not work as root, especially not while editing files on a non-root Splunk, it will bring you into trouble.
I'd suggest using a normal user, you could use the user Splunk is running at.

[splunklearner]

what to give instead of sudo -i while logging in? Please let me know @kiran_panchavat 

[kiran_panchavat]

@splunklearner 

Instead of using sudo -i, which logs you in as the root user, you should use the following command to switch to the Splunk user

sudo su - splunk

[kiran_panchavat]

@splunklearner 

1st stop splunk and after that sudo chown -R splunk:splunk /opt/splunk. Then start it again.

In your example, you are trying to do that initialization as root. Only time when you should use user root is to enable boot-start (or start/stop/restart with systemd). 

The error message shows that for some reason it tried to write some status information to /root/.splunk directory which didn't succeed as splunk is running as user splunk not as root.

You should do sudo -u splunk then run those commands as user splunk. 

/opt/splunk/bin/splunk reload deploy-server