Splunk internal logs stopped logging; can you please provide information about permission setup for log files?

[dbanerjee17]

Hello Splunkers:
This question is for the splunkers who are running their instances with splunk user.
Three logs have stopped logging since April 17, 2018. They are splunkd.log, metrics.log and splunk-access.log. Also, now I am getting access denied error for splunkd.log while restarting splunk and also KV store failed due to access denied error for /mnt/data/splunk/kvstore/mongo/_tmp directory. Our splunk instance runs with splunk user. On 17th, I, by mistake started with root user. After realizing it, I stopped and restarted splunk with splunk user. Just after that, the problem started. All the three logs files and_tmp directory mentioned above are currently running with root. All others are running with splunk user all other files have splunk ownership. I am not sure whether the above mentioned log files and _tmp direct should be with splunk ownership. I have struggling with issue for the past few days. Any help on this will be highly appreciated. Can you please check your instances and let me know the ownership of the splunkd.log, metrics.log, splunk-access.log and /mnt/data/splunk/kvstore/mongo/_tmp directory?
Thank you in advance.
Deb

[FrankVl]

When you accidentally started it as root, several files will have changed ownership to root. When switching back to splunk user, the splunk process lost access to those files.

What you should have done (and should now do) is:

1. stop splunk
2. set the ownership of the entire splunk folder (and other locations it uses) to the splunk user and its group
3. start splunk as splunk user

To prevent this issue in the future:

1. go to splunk/etc/
2. mv splunk-launch.conf.default splunk-launch.conf
3. on the bottom line activate the setting: SPLUNK_OS_USER = splunk

This will force splunk to always start as that user, regardless of which account you use to execute the splunk start command.

[janispelss]

As far as I'm aware, all files and directories in your Splunk installation path should be owned by the same user that's used for running Splunk. And it's definitely the case for the files you mentioned.