Splunk unable to read specific files even though it can when started from the command line

[bosburn_splunk]

Scenario:

Running Splunk 6.1.x (either the forwarder or indexer)

Using /etc/init.d/splunk to start/stop Splunk as user splunk (i.e. ./splunk enable boot-start -user splunk)

In order to read root owned files and keep Splunk running as non-root, I've chowned certain files as root:splunk. For example:

-rw-r----- 1 root splunk  3726 Aug  8 07:01 cron
-rw-r----- 1 root splunk 25850 Aug  8 07:41 messages

I can read the files from the command line just fine. But Splunk can't read them - I'm getting the following errors:

08-08-2014 06:40:54.703 -0700 WARN TailingProcessor - Insufficient permissions to read file='/var/log/cron' (hint: Permission denied). 
08-08-2014 06:40:54.704 -0700 WARN TailingProcessor - Insufficient permissions to read file='/var/log/messages' (hint: Permission denied). 

What gives?

[fharding]

Per Kyle from support's recommendation, I modified the init script to use /bin/su like in the older 6.0.x scripts and it looks like it's fixed the issue as a stop-gap solution. I am running Splunk Enterprise 6.1.3.

#!/bin/sh
#
# /etc/init.d/splunk
# init script for Splunk.
# generated by 'splunk enable boot-start'.
#
# chkconfig: 2345 90 60
# description: Splunk indexer service
#
RETVAL=0

. /etc/init.d/functions

splunk_start() {
  echo Starting Splunk...
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" start --no-prompt --answer-yes"
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_stop() {
  echo Stopping Splunk...
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" stop "
  RETVAL=$?
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/splunk
}
splunk_restart() {
  echo Restarting Splunk...
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" restart "
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_status() {
  echo Splunk status:
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" status "
  RETVAL=$?
}
case "$1" in
  start)
    splunk_start
    ;;
  stop)
    splunk_stop
    ;;
  restart)
    splunk_restart
    ;;
  status)
    splunk_status
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac

exit $RETVAL

[kevbod]

Guys , is this post relevant to the eval version of Enterprise 6.2.2 Build 255606 ? I am running on redhat 6.5 ([root@splunk01 ~]# uname -a - RESULT IS- Linux splunk01 2.6.32-431.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64 x86_64 x86_64 GNU/Linux.)

Install file is splunk-6.2.3-264376-linux-2.6-x86_64.rpm

I installed the enterprise version of splunk as root, but ran into problems with the configuration of the splunk add on for unix looping and never saying successful. A post here on the forum suggested that this was a permissions issue and i needed to install as a non-root user so that is what i did.

I have fixed the splunk script as stated above and I still get a mass of permissions problems that i frankly am baffled by. This occurs if installing as root, splunker (a user i created) and splunkcreated by the rpm install

[splunk@splunk01 ~]$ /bin/su - splunk -c "\"/opt/splunk/bin/splunk\" start --no-prompt --answer-yes"
Password:
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

Splunk> Now with more code!

Checking prerequisites...
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied
Checking http port [8000]: Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking mgmt port [8089]: Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking appserver port [127.0.0.1:8065]: Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
ERROR - Failed opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
Checking configuration... Error while parsing '/opt/splunk/etc/myinstall/splunkd.xml':
[Errno 13] Permission denied: '/opt/splunk/etc/myinstall/splunkd.xml'

There were problems with the configuration files.
Would you like to ignore these errors? [y/n]: y
Done.
Creating: /opt/splunk/var/run/splunk/appserver/i18n
Warning: cannot create "/opt/splunk/var/run/splunk/appserver/i18n"
[splunk@splunk01 ~]$

Regards Kevin

[bosburn_splunk · Accepted solution]

A bug has been identified (SPL-87651) where using /etc/init.d/splunk that has been set up by ./splunk enable boot-start -user splunk isn't applying the right GID when accessing files.

This can be worked around as starting Splunk using the user splunk and $SPLUNK_HOME/bin/splunk start vs. /etc/init.d/splunk.

Brian

[blebit]

Hi,
does it work with other agent versions ?
thanks

[BARNEYRUDD]

I'm on 7.3.1 (universal forwarder) and have this problem. The workaround proposed below by @fharding worked for me.

[ccglanville]

Hi Brian,

Do you know when a patch will be released? Is it a simple modification to the /etc/init.d script? I see that the 6.1.x script is missing "/bin/su - splunk -c".

As far as the workaround goes - I assume it won't survive a server reboot?

I also noticed that the files splunk it attempting to read need the splunk user to at least have standard read granted. The ACL access is not enough.

thanks,
Chris