topic Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0? in Installation

Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

BongoTheWhippet — Thu, 11 Jun 2020 21:46:12 GMT

I've gone through the answers here and tried the following:

Unlocking stale PIDs
clean locks
chown -R <user>:<group> /opt/splunk

But nothing seems to work. The last message:

Please login as an administrator and correct issue.

When I'm root can only mean that something is hardcoded that really shouldn't be.

Here's the complete output (which is the same if I run it under the splunk user, or the user set in the /etc/init.d/splunk script which is irrelevant here I think) :

root@ubuntu:/opt/splunk/bin# ./splunk start

    Splunk> All batbelt. No tights.

    Checking prerequisites...
        Checking http port [8000]: open
        Checking mgmt port [8089]: open
        Checking configuration...  Done.
        Checking critical directories...    Done
        Checking indexes...
            Validated: _audit _blocksignature _internal _introspection _thefishbucket history main snort_test summary test
        Done
    Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
    Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
    Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
    Error opening username mapping file: /opt/splunk/etc/users/users.ini
    Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
    New certs have been generated in '/opt/splunk/etc/auth'.
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
    ERROR - Error opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied


    Your active group is invalid. Please login as an administrator and correct issue.

    ERROR IniFile - Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
    ERROR UsernameMapper - Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
    ERROR IniFile - Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
    ERROR UsernameMapper - Error opening username mapping file: /opt/splunk/etc/users/users.ini
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/splunk_app_for_nix/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
    ERROR IniFile - Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
    ERROR UsernameMapper - Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
    ERROR IniFile - Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
    ERROR UsernameMapper - Error opening username mapping file: /opt/splunk/etc/users/users.ini
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/launcher/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/search/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/splunk_app_for_nix/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
    ERROR ConfObjectManagerDB - Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
    The SPLUNK_DB environment variable was defined but the test file ("/opt/splunk/var/lib/splunk/test.kMgOmj") could not be created by the current user: Permission denied
    Locking test failed on filesystem in path /opt/splunk/var/lib/splunk with code '7'.  Please file a case online at http://www.splunk.com/page/submit_issue
        Checking filesystem compatibility...  root@ubuntu:/opt/splunk/bin#

Any ideas anyone? Thanks and regards

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

richgalloway — Sun, 22 Mar 2015 21:22:10 GMT

Would you please show us the permissions for /opt/splunk/etc/system/local/server.conf and its parent directories?
Are you running SELinux?

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

BongoTheWhippet — Sun, 22 Mar 2015 21:53:09 GMT

Hi. No SELinux here (shudders!)

ls -l /opt/splunk/etc/system/local/server.conf
-rw------- 1 splunk splunk 527 Nov 29 18:49 /opt/splunk/etc/system/local/server.conf

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

BongoTheWhippet — Sun, 22 Mar 2015 22:00:27 GMT

splunk@ubuntu:~/bin$ ls -l /opt/splunk/etc/system/local
total 32
-rw------- 1 splunk splunk   0 Mar 22 18:47 eventtypes.conf
-rw-r--r-- 1 splunk splunk  80 Mar 22 18:47 indexes.conf
-rw-r--r-- 1 splunk splunk  80 Mar 22 18:47 indexes.conf.old
-rw------- 1 splunk splunk  24 Mar 22 18:47 inputs.conf
-rw------- 1 splunk splunk  48 Sep 16  2014 limits.conf
-rw------- 1 splunk splunk 261 Mar 22 18:47 migration.conf
-r--r--r-- 1 splunk splunk 265 Jul 30  2014 README
-rw------- 1 splunk splunk   0 Nov 29 16:10 serverclass.conf
-rw------- 1 splunk splunk 527 Nov 29 18:49 server.conf
-rw------- 1 splunk splunk  34 Feb 13 15:36 web.conf
splunk@ubuntu:~/bin$ ls -l /opt/splunk/etc/system
total 44
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:46 bin
drwxr-xr-x 3 splunk splunk  4096 Mar 22 18:46 default
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:47 local
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:46 lookups
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:46 metadata
drwxr-xr-x 2 splunk splunk 20480 Mar 22 18:46 README
drwxr-xr-x 2 splunk splunk  4096 Mar 22 18:46 static

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

BongoTheWhippet — Sun, 22 Mar 2015 22:05:00 GMT

The board won't let me post the remaining lines presumably because it's misinterpreting a string in the output as code. But its much of the same. I've not changed any permissions between upgrading so anything that looks odd is an output of the upgrade.

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

bjoernjensen — Sun, 22 Mar 2015 22:58:39 GMT

Looks to me that it should work .. just to make sure: Have you had a look into the known issues for 6.2.0, SPL-89640 respectively? Could you post an ls -l $SPLUNK_HOME/var/log/introspection
http://docs.splunk.com/Documentation/Splunk/6.2.0/ReleaseNotes/KnownIssues

If you run Splunk Enterprise on Linux as a non-root user, and use an RPM to upgrade, the RPM writes the $SPLUNK_HOME/var/log/introspection directory as root. This can cause errors when you attempt to start the instance later. To prevent this, chown the $SPLUNK_HOME/var/log/introspection directory to the user Splunk Enterprise runs as after upgrading and before restarting Splunk Enterprise.

All the best - B

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

edavson — Tue, 24 Mar 2015 13:14:07 GMT

Check if FIPS is enabled in /splunk/etc/splunk-launch.conf. I believe it is enabled by default.

you can disable by adding:

SPLUNK_FIPS=0

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

matthieu_araman — Tue, 24 Mar 2015 13:45:36 GMT

Hmm, you're giving files to splunk but you are starting splunk as root ... doesn't seem logic to me.

I'm assuming you wan't to make splunk run as splunk user and configured it accordingly initially.

can you try :

grep splunk /etc/passwd -> check splunk user exist, note group (assuming splunk)
grep splunk /etc/group -> check splunk group exist (because the error message looks like there's a problem with the group)
make sure splunk is not running
as root :
chown -R splunk. /opt/splunk (give all the file in /opt/splunk to splunk user with the default group of splunk user) as splunk user (not root) /opt/splunk/bin/splunk start check for errors

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

BongoTheWhippet — Sat, 28 Mar 2015 20:26:37 GMT

It appears that install as a .deb does the same thing. The complication is that I have no users.ini file because this question is a free home installation .

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

BongoTheWhippet — Sat, 28 Mar 2015 20:30:31 GMT

Starting it as root should solve the problem, logically. But it doesn't. There's something hardcoded in there that requires the installation user to start it under that context only.

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

fabioportes — Mon, 28 Sep 2020 20:12:46 GMT

I just upgraded from 6.2.1 to 6.2.3 using DEB packages.
$SPLUNK_HOME/etc/splunk-launch.conf had:

SPLUNK_OS_USER=splunker

My OS user (used by enable boot-start) is 'splunk', so I changed it and magic happened.

I hope it helps somebody.

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

ashokqos — Tue, 29 Sep 2020 09:24:30 GMT

Thanks. I upgraded from 6.2.2 to 6.3.3 on ubuntu (deb package). When I tried splunk start I got permission errors.
Then I changed the SPLUNK_OS_USER from sadmin to splunk in /opt/splunk/etc/splunk-launch.conf. Now /opt/splunk/bin/splunk start worked.

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

athorat — Wed, 13 Jul 2016 18:42:48 GMT

@fabioportes
Thanks it did help (Y)

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

rphillips_splk — Tue, 29 Sep 2020 10:28:14 GMT

check the splunk-launch.conf in ($SPLUNK_HOME/etc/splunk-launch.conf) and see if the SPLUNK_OS_USER variable is set.

`# If SPLUNK_OS_USER is set, then Splunk service will only start

if the 'splunk [re]start [splunkd]' command is invoked by a user who

is, or can effectively become via setuid(2), $SPLUNK_OS_USER.

(This setting can be specified as username or as UID.)

SPLUNK_OS_USER

SPLUNK_OS_USER=splunk`

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

sutanunandigram — Sat, 27 Jul 2019 01:06:49 GMT

One thing helped me . previously i configured boot-start with user splunk.

./splunk enable boot-start -user splunk

So i changed it to root. And the issue resolved.

./splunk enable boot-start -user root

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

sutanunandigram — Sat, 27 Jul 2019 01:08:06 GMT

One thing helped me . previously i configured boot-start with user splunk.

./splunk enable boot-start -user splunk

So i changed it to root. And the issue resolved.

./splunk enable boot-start -user root

Re: Why is Splunk not starting on Linux with permission denied errors after upgrading to 6.2.0?

richgalloway — Sat, 27 Jul 2019 15:26:27 GMT

One should not run Splunk (or anything not part of the OS) as root.