Facing issue after changing the Root user as non Root user in /opt/splunk folder on shard search head with pooling config

[Hemnaath]

Hi All, Can any one guide me on how to fix this issue, after changing the Root user as non Root user "su splunk" I am getting an error in splunkd.log, I am stuck with this in my prod environment.

I have followed the below steps to run as noon root user.

sudo su -
cd /opt/splunk/bin
./splunk stop
chown -R splunk:splunk /opt/splunk
su splunk
cd /opt/splunk/bin
./splunk start

After the change, I could the splunk started with some errors and it continued to be running with splunk user but when check in the splunkd .log I can see Error message

Splunkd .log error info
12-28-2016 13:12:28.366 -0500 ERROR DispatchReaper - Failed to reap /splunk_search_pool/var/run/splunk/dispatch/rt_scheduler_hflgg900searchRMD545c376f36cb02e56_at_1464419460_1224.404406 because of Permission denied
12-28-2016 13:12:28.366 -0500 ERROR DispatchReaper - Failed to reap /splunk_search_pool/var/run/splunk/dispatch/rt_schedulerhflgg900search_RMD545c376f36cb02e56_at_1464419460_1224.446500 because of Permission denied
12-28-2016 13:12:29.697 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/sideview_utils/metadata/local.meta: Permission denied
12-28-2016 13:12:30.290 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/learned/metadata/local.meta: Permission denied
12-28-2016 13:12:30.291 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/user-prefs/metadata/local.meta: Permission denied
12-28-2016 13:12:31.813 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/search/metadata/local.meta: Permission denied
12-28-2016 13:12:33.887 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/sideview_utils/metadata/local.meta: Permission denied

Kindly guide on how to fix this issue.

thanks in advance.

[jkat54]

Try this chown command instead

chown -Rf splunk. /opt/splunk

Same other steps though.

[Hemnaath]

thanks Jkat for your inputs, anyway I agree with you to you and I know its very risk to run splunk as root, but need to get an approval to change this in prod.

[jkat54]

Also it's a huge security risk and against core best practices to run Splunk as root. You should stop doing that ASAP.

[jkat54]

9 times out of 10, you should use this command as root instead of just editing splunk-launch.conf

$SPLUNK_HOME/bin/splunk enable boot-start -user [username]

Where [username] = whatever linux user name you've decided to run splunk as.

And I stand by 'chown -Rf splunk. /path/to/splunk' versus chown -R

[Hemnaath]

Hey Jkat54, thanks for your input but I have restart the splunk with root user as other splunk instance is using the same user id and now I could see the is no permission denied error as I am using the root user id over splunk folder.

But anyway these are the steps should be followed to change splunk from root user id to non root id, i.e. run splunk with splunk user id:

1) sudo su -
2) with root user id stop the splunk service
3) With root user id change the permission of the splunk folder to use splunk user id
chown -R splunk:splunk /opt/splunk
4) Add this stanza " SPLUNK_OS_USER=splunk" in /opt/splunk/etc/splunk-launch.conf and save it.
5) switch to splunk user id using su splunk
6) Try to start the splunk using splunk user id.
7) check the splunk process using which id by using command ps -ef | grep splunkd

Note: If you see any permission problem try to identify the folder which is having permission issue by going to the folder,check for the folder permission by using ls -ll command. If you find that particular folder is used by root change it splunk to that folder alone.

[jkat54]

If out doesn't then ill owe you a beer.

That's the process I follow every time. Only way it could be different is if for some reason the file system they are on are read only or otherwise not write able by root.

[skoelpin]

@jkat54 is correct on this one, but I wouldn't test in production if you are unsure.. Best case scenario here is you get a free beer

[Hemnaath]

thanks Jkat54, but now I am little bit concern on executing commands as it is in Production environment.
By doing it force fully will it fix the issue.

thanks in advance.