Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

[BongoTheWhippet]

I've gone through the answers here and tried the following:

1. Unlocking stale PIDs
2. clean locks
3. chown -R <user>:<group> /opt/splunk

But nothing seems to work. The last message:

Please login as an administrator and correct issue.

When I'm root can only mean that something is hardcoded that really shouldn't be.

Here's the complete output (which is the same if I run it under the splunk user, or the user set in the /etc/init.d/splunk script which is irrelevant here I think) :

root@ubuntu:/opt/splunk/bin# ./splunk start

    Splunk> All batbelt. No tights.

    Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking configuration...  Done.
        Checking critical directories...    Done
        Checking indexes...
            Validated: _audit _blocksignature _internal _introspection _thefishbucket history main snort_test summary test
        Done
    Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
    Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
    Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
    Error opening username mapping file: /opt/splunk/etc/users/users.ini
    Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
    New certs have been generated in '/opt/splunk/etc/auth'.
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied


    Your active group is invalid. Please login as an administrator and correct issue.

    ERROR IniFile - Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
    ERROR UsernameMapper - Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
    ERROR IniFile - Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
    ERROR UsernameMapper - Error opening username mapping file: /opt/splunk/etc/users/users.ini
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/splunk_app_for_nix/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
    ERROR IniFile - Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
    ERROR UsernameMapper - Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
    ERROR IniFile - Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
    ERROR UsernameMapper - Error opening username mapping file: /opt/splunk/etc/users/users.ini
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/splunk_app_for_nix/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
    The SPLUNK_DB environment variable was defined but the test file ("/opt/splunk/var/lib/splunk/test.kMgOmj") could not be created by the current user: Permission denied
    Locking test failed on filesystem in path /opt/splunk/var/lib/splunk with code '7'.  Please file a case online at http://www.splunk.com/page/submit_issue
        Checking filesystem compatibility...  root@ubuntu:/opt/splunk/bin# 

Any ideas anyone? Thanks and regards

[sutanunandigram]

One thing helped me . previously i configured boot-start with user splunk.

./splunk enable boot-start -user splunk

So i changed it to root. And the issue resolved.

./splunk enable boot-start -user root

[richgalloway]

One should not run Splunk (or anything not part of the OS) as root.

[sutanunandigram]

One thing helped me . previously i configured boot-start with user splunk.

./splunk enable boot-start -user splunk

So i changed it to root. And the issue resolved.

./splunk enable boot-start -user root

[rphillips_splk]

check the splunk-launch.conf in ($SPLUNK_HOME/etc/splunk-launch.conf) and see if the SPLUNK_OS_USER variable is set.

`# If SPLUNK_OS_USER is set, then Splunk service will only start

if the 'splunk [re]start [splunkd]' command is invoked by a user who

is, or can effectively become via setuid(2), $SPLUNK_OS_USER.

(This setting can be specified as username or as UID.)

SPLUNK_OS_USER

SPLUNK_OS_USER=splunk`

[fabioportes]

I just upgraded from 6.2.1 to 6.2.3 using DEB packages.
$SPLUNK_HOME/etc/splunk-launch.conf had:

SPLUNK_OS_USER=splunker

My OS user (used by enable boot-start) is 'splunk', so I changed it and magic happened.

I hope it helps somebody.

[ashokqos]

Thanks. I upgraded from 6.2.2 to 6.3.3 on ubuntu (deb package). When I tried splunk start I got permission errors.
Then I changed the SPLUNK_OS_USER from sadmin to splunk in /opt/splunk/etc/splunk-launch.conf. Now /opt/splunk/bin/splunk start worked.

[athorat]

@fabioportes
Thanks it did help (Y)

[matthieu_araman]

Hmm, you're giving files to splunk but you are starting splunk as root ... doesn't seem logic to me.

I'm assuming you wan't to make splunk run as splunk user and configured it accordingly initially.

can you try :

• grep splunk /etc/passwd -> check splunk user exist, note group (assuming splunk)
• grep splunk /etc/group -> check splunk group exist (because the error message looks like there's a problem with the group)
• make sure splunk is not running
• as root :
• chown -R splunk. /opt/splunk (give all the file in /opt/splunk to splunk user with the default group of splunk user) as splunk user (not root) /opt/splunk/bin/splunk start check for errors

[BongoTheWhippet]

Starting it as root should solve the problem, logically. But it doesn't. There's something hardcoded in there that requires the installation user to start it under that context only.

[edavson]

Check if FIPS is enabled in /splunk/etc/splunk-launch.conf. I believe it is enabled by default.

you can disable by adding:

SPLUNK_FIPS=0

[bjoernjensen · Accepted solution]

Looks to me that it should work .. just to make sure: Have you had a look into the known issues for 6.2.0, SPL-89640 respectively? Could you post an ls -l $SPLUNK_HOME/var/log/introspection
http://docs.splunk.com/Documentation/Splunk/6.2.0/ReleaseNotes/KnownIssues

If you run Splunk Enterprise on Linux as a non-root user, and use an RPM to upgrade, the RPM writes the $SPLUNK_HOME/var/log/introspection directory as root. This can cause errors when you attempt to start the instance later. To prevent this, chown the $SPLUNK_HOME/var/log/introspection directory to the user Splunk Enterprise runs as after upgrading and before restarting Splunk Enterprise.

All the best - B

[BongoTheWhippet]

It appears that install as a .deb does the same thing. The complication is that I have no users.ini file because this question is a free home installation .

[richgalloway]

Would you please show us the permissions for /opt/splunk/etc/system/local/server.conf and its parent directories?
Are you running SELinux?

[BongoTheWhippet]

Hi. No SELinux here (shudders!)

ls -l /opt/splunk/etc/system/local/server.conf
-rw------- 1 splunk splunk 527 Nov 29 18:49 /opt/splunk/etc/system/local/server.conf

[BongoTheWhippet]

splunk@ubuntu:~/bin$ ls -l /opt/splunk/etc/system/local
total 32
-rw------- 1 splunk splunk   0 Mar 22 18:47 eventtypes.conf
-rw-r--r-- 1 splunk splunk  80 Mar 22 18:47 indexes.conf
-rw-r--r-- 1 splunk splunk  80 Mar 22 18:47 indexes.conf.old
-rw------- 1 splunk splunk  24 Mar 22 18:47 inputs.conf
-rw------- 1 splunk splunk  48 Sep 16  2014 limits.conf
-rw------- 1 splunk splunk 261 Mar 22 18:47 migration.conf
-r--r--r-- 1 splunk splunk 265 Jul 30  2014 README
-rw------- 1 splunk splunk   0 Nov 29 16:10 serverclass.conf
-rw------- 1 splunk splunk 527 Nov 29 18:49 server.conf
-rw------- 1 splunk splunk  34 Feb 13 15:36 web.conf
splunk@ubuntu:~/bin$ ls -l /opt/splunk/etc/system
total 44
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:46 bin
drwxr-xr-x 3 splunk splunk  4096 Mar 22 18:46 default
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:47 local
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:46 lookups
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:46 metadata
drwxr-xr-x 2 splunk splunk 20480 Mar 22 18:46 README
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:46 static

[BongoTheWhippet]

The board won't let me post the remaining lines presumably because it's misinterpreting a string in the output as code. But its much of the same. I've not changed any permissions between upgrading so anything that looks odd is an output of the upgrade.