topic Re: Splunk unable to read specific files even though it can when started from the command line in Security

Splunk unable to read specific files even though it can when started from the command line

bosburn_splunk — Fri, 08 Aug 2014 11:54:54 GMT

Scenario:

Running Splunk 6.1.x (either the forwarder or indexer)

Using /etc/init.d/splunk to start/stop Splunk as user splunk (i.e. ./splunk enable boot-start -user splunk)

In order to read root owned files and keep Splunk running as non-root, I've chowned certain files as root:splunk. For example:

-rw-r----- 1 root splunk  3726 Aug  8 07:01 cron
-rw-r----- 1 root splunk 25850 Aug  8 07:41 messages

I can read the files from the command line just fine. But Splunk can't read them - I'm getting the following errors:

08-08-2014 06:40:54.703 -0700 WARN TailingProcessor - Insufficient permissions to read file='/var/log/cron' (hint: Permission denied). 
08-08-2014 06:40:54.704 -0700 WARN TailingProcessor - Insufficient permissions to read file='/var/log/messages' (hint: Permission denied).

What gives?

Re: Splunk unable to read specific files even though it can when started from the command line

bosburn_splunk — Fri, 08 Aug 2014 11:55:13 GMT

A bug has been identified (SPL-87651) where using /etc/init.d/splunk that has been set up by ./splunk enable boot-start -user splunk isn't applying the right GID when accessing files.

This can be worked around as starting Splunk using the user splunk and $SPLUNK_HOME/bin/splunk start vs. /etc/init.d/splunk.

Brian

Re: Splunk unable to read specific files even though it can when started from the command line

ccglanville — Mon, 25 Aug 2014 21:09:57 GMT

Hi Brian,

Do you know when a patch will be released? Is it a simple modification to the /etc/init.d script? I see that the 6.1.x script is missing "/bin/su - splunk -c".

As far as the workaround goes - I assume it won't survive a server reboot?

I also noticed that the files splunk it attempting to read need the splunk user to at least have standard read granted. The ACL access is not enough.

thanks,
Chris

Re: Splunk unable to read specific files even though it can when started from the command line

fharding — Tue, 09 Sep 2014 00:20:47 GMT

Per Kyle from support's recommendation, I modified the init script to use /bin/su like in the older 6.0.x scripts and it looks like it's fixed the issue as a stop-gap solution. I am running Splunk Enterprise 6.1.3.

#!/bin/sh
#
# /etc/init.d/splunk
# init script for Splunk.
# generated by 'splunk enable boot-start'.
#
# chkconfig: 2345 90 60
# description: Splunk indexer service
#
RETVAL=0

. /etc/init.d/functions

splunk_start() {
  echo Starting Splunk...
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" start --no-prompt --answer-yes"
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_stop() {
  echo Stopping Splunk...
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" stop "
  RETVAL=$?
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/splunk
}
splunk_restart() {
  echo Restarting Splunk...
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" restart "
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_status() {
  echo Splunk status:
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" status "
  RETVAL=$?
}
case "$1" in
  start)
    splunk_start
    ;;
  stop)
    splunk_stop
    ;;
  restart)
    splunk_restart
    ;;
  status)
    splunk_status
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac

exit $RETVAL

Re: Splunk unable to read specific files even though it can when started from the command line

blebit — Tue, 24 Mar 2015 15:46:28 GMT

Hi,
does it work with other agent versions ?
thanks

Re: Splunk unable to read specific files even though it can when started from the command line

kevbod — Tue, 29 Sep 2020 20:13:18 GMT

Guys , is this post relevant to the eval version of Enterprise 6.2.2 Build 255606 ? I am running on redhat 6.5 ([root@splunk01 ~]# uname -a - RESULT IS- Linux splunk01 2.6.32-431.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64 x86_64 x86_64 GNU/Linux.)

Install file is splunk-6.2.3-264376-linux-2.6-x86_64.rpm

I installed the enterprise version of splunk as root, but ran into problems with the configuration of the splunk add on for unix looping and never saying successful. A post here on the forum suggested that this was a permissions issue and i needed to install as a non-root user so that is what i did.

I have fixed the splunk script as stated above and I still get a mass of permissions problems that i frankly am baffled by. This occurs if installing as root, splunker (a user i created) and splunkcreated by the rpm install

[splunk@splunk01 ~]$ /bin/su - splunk -c "\"/opt/splunk/bin/splunk\" start --no-prompt --answer-yes"
Password:
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

Splunk> Now with more code!

Checking prerequisites...
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied
Checking http port [8000]: Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking mgmt port [8089]: Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking appserver port [127.0.0.1:8065]: Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
ERROR - Failed opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
Checking configuration... Error while parsing '/opt/splunk/etc/myinstall/splunkd.xml':
[Errno 13] Permission denied: '/opt/splunk/etc/myinstall/splunkd.xml'

There were problems with the configuration files.
Would you like to ignore these errors? [y/n]: y
Done.
Creating: /opt/splunk/var/run/splunk/appserver/i18n
Warning: cannot create "/opt/splunk/var/run/splunk/appserver/i18n"
[splunk@splunk01 ~]$

Regards Kevin

Re: Splunk unable to read specific files even though it can when started from the command line

BARNEYRUDD — Tue, 17 Sep 2019 10:03:05 GMT

I'm on 7.3.1 (universal forwarder) and have this problem. The workaround proposed below by @fharding worked for me.