Security

Facing issue after changing the Root user as non Root user in /opt/splunk folder on shard search head with pooling config

Hemnaath
Motivator

Hi All, Can any one guide me on how to fix this issue, after changing the Root user as non Root user "su splunk" I am getting an error in splunkd.log, I am stuck with this in my prod environment.

I have followed the below steps to run as noon root user.

sudo su -
cd /opt/splunk/bin
./splunk stop
chown -R splunk:splunk /opt/splunk
su splunk
cd /opt/splunk/bin
./splunk start

After the change, I could the splunk started with some errors and it continued to be running with splunk user but when check in the splunkd .log I can see Error message

Splunkd .log error info
12-28-2016 13:12:28.366 -0500 ERROR DispatchReaper - Failed to reap /splunk_search_pool/var/run/splunk/dispatch/rt_scheduler_hflgg900searchRMD545c376f36cb02e56_at_1464419460_1224.404406 because of Permission denied
12-28-2016 13:12:28.366 -0500 ERROR DispatchReaper - Failed to reap /splunk_search_pool/var/run/splunk/dispatch/rt_scheduler
hflgg900search_RMD545c376f36cb02e56_at_1464419460_1224.446500 because of Permission denied
12-28-2016 13:12:29.697 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/sideview_utils/metadata/local.meta: Permission denied
12-28-2016 13:12:30.290 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/learned/metadata/local.meta: Permission denied
12-28-2016 13:12:30.291 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/user-prefs/metadata/local.meta: Permission denied
12-28-2016 13:12:31.813 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/search/metadata/local.meta: Permission denied
12-28-2016 13:12:33.887 -0500 ERROR ConfObjectManagerDB - Cannot initialize: /splunk_search_pool/etc/apps/sideview_utils/metadata/local.meta: Permission denied

Kindly guide on how to fix this issue.

thanks in advance.

Tags (1)
0 Karma

jkat54
SplunkTrust
SplunkTrust

Try this chown command instead

chown -Rf splunk. /opt/splunk

Same other steps though.

0 Karma

Hemnaath
Motivator

thanks Jkat for your inputs, anyway I agree with you to you and I know its very risk to run splunk as root, but need to get an approval to change this in prod.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Also it's a huge security risk and against core best practices to run Splunk as root. You should stop doing that ASAP.

0 Karma

jkat54
SplunkTrust
SplunkTrust

9 times out of 10, you should use this command as root instead of just editing splunk-launch.conf

$SPLUNK_HOME/bin/splunk enable boot-start -user [username]

Where [username] = whatever linux user name you've decided to run splunk as.

And I stand by 'chown -Rf splunk. /path/to/splunk' versus chown -R

0 Karma

Hemnaath
Motivator

Hey Jkat54, thanks for your input but I have restart the splunk with root user as other splunk instance is using the same user id and now I could see the is no permission denied error as I am using the root user id over splunk folder.

But anyway these are the steps should be followed to change splunk from root user id to non root id, i.e. run splunk with splunk user id:

1) sudo su -
2) with root user id stop the splunk service
3) With root user id change the permission of the splunk folder to use splunk user id
chown -R splunk:splunk /opt/splunk
4) Add this stanza " SPLUNK_OS_USER=splunk" in /opt/splunk/etc/splunk-launch.conf and save it.
5) switch to splunk user id using su splunk
6) Try to start the splunk using splunk user id.
7) check the splunk process using which id by using command ps -ef | grep splunkd

Note: If you see any permission problem try to identify the folder which is having permission issue by going to the folder,check for the folder permission by using ls -ll command. If you find that particular folder is used by root change it splunk to that folder alone.

0 Karma

jkat54
SplunkTrust
SplunkTrust

If out doesn't then ill owe you a beer.

That's the process I follow every time. Only way it could be different is if for some reason the file system they are on are read only or otherwise not write able by root.

skoelpin
SplunkTrust
SplunkTrust

@jkat54 is correct on this one, but I wouldn't test in production if you are unsure.. Best case scenario here is you get a free beer

0 Karma

Hemnaath
Motivator

thanks Jkat54, but now I am little bit concern on executing commands as it is in Production environment.
By doing it force fully will it fix the issue.

thanks in advance.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...