Security

Splunk unable to read specific files even though it can when started from the command line

bosburn_splunk
Splunk Employee
Splunk Employee

Scenario:

Running Splunk 6.1.x (either the forwarder or indexer)

Using /etc/init.d/splunk to start/stop Splunk as user splunk (i.e. ./splunk enable boot-start -user splunk)

In order to read root owned files and keep Splunk running as non-root, I've chowned certain files as root:splunk. For example:

-rw-r----- 1 root splunk  3726 Aug  8 07:01 cron
-rw-r----- 1 root splunk 25850 Aug  8 07:41 messages

I can read the files from the command line just fine. But Splunk can't read them - I'm getting the following errors:

08-08-2014 06:40:54.703 -0700 WARN TailingProcessor - Insufficient permissions to read file='/var/log/cron' (hint: Permission denied). 
08-08-2014 06:40:54.704 -0700 WARN TailingProcessor - Insufficient permissions to read file='/var/log/messages' (hint: Permission denied). 

What gives?

1 Solution

bosburn_splunk
Splunk Employee
Splunk Employee

A bug has been identified (SPL-87651) where using /etc/init.d/splunk that has been set up by ./splunk enable boot-start -user splunk isn't applying the right GID when accessing files.

This can be worked around as starting Splunk using the user splunk and $SPLUNK_HOME/bin/splunk start vs. /etc/init.d/splunk.

Brian

View solution in original post

fharding
Explorer

Per Kyle from support's recommendation, I modified the init script to use /bin/su like in the older 6.0.x scripts and it looks like it's fixed the issue as a stop-gap solution. I am running Splunk Enterprise 6.1.3.

#!/bin/sh
#
# /etc/init.d/splunk
# init script for Splunk.
# generated by 'splunk enable boot-start'.
#
# chkconfig: 2345 90 60
# description: Splunk indexer service
#
RETVAL=0

. /etc/init.d/functions

splunk_start() {
  echo Starting Splunk...
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" start --no-prompt --answer-yes"
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_stop() {
  echo Stopping Splunk...
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" stop "
  RETVAL=$?
  [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/splunk
}
splunk_restart() {
  echo Restarting Splunk...
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" restart "
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}
splunk_status() {
  echo Splunk status:
  /bin/su - splunk -c "\"/opt/splunkforwarder/bin/splunk\" status "
  RETVAL=$?
}
case "$1" in
  start)
    splunk_start
    ;;
  stop)
    splunk_stop
    ;;
  restart)
    splunk_restart
    ;;
  status)
    splunk_status
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    exit 1
    ;;
esac

exit $RETVAL

kevbod
New Member

Guys , is this post relevant to the eval version of Enterprise 6.2.2 Build 255606 ? I am running on redhat 6.5 ([root@splunk01 ~]# uname -a - RESULT IS- Linux splunk01 2.6.32-431.el6.x86_64 #1 SMP Sun Nov 10 22:19:54 EST 2013 x86_64 x86_64 x86_64 GNU/Linux.)

Install file is splunk-6.2.3-264376-linux-2.6-x86_64.rpm

I installed the enterprise version of splunk as root, but ran into problems with the configuration of the splunk add on for unix looping and never saying successful. A post here on the forum suggested that this was a permissions issue and i needed to install as a non-root user so that is what i did.

I have fixed the splunk script as stated above and I still get a mass of permissions problems that i frankly am baffled by. This occurs if installing as root, splunker (a user i created) and splunkcreated by the rpm install

[splunk@splunk01 ~]$ /bin/su - splunk -c "\"/opt/splunk/bin/splunk\" start --no-prompt --answer-yes"
Password:
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkd.pid" unreadable.: Permission denied

Splunk> Now with more code!

Checking prerequisites...
Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
Pid file "/opt/splunk/var/run/splunk/splunkweb.pid" unreadable.: Permission denied
Checking http port [8000]: Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking mgmt port [8089]: Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
Checking appserver port [127.0.0.1:8065]: Cannot open file=/opt/splunk/etc/system/local/server.conf for parsing: Permission denied
Cannot create username mapping file: /opt/splunk/etc/users/users.ini: Permission denied
Cannot open file=/opt/splunk/etc/users/users.ini for parsing: Permission denied
Error opening username mapping file: /opt/splunk/etc/users/users.ini
Cannot initialize: /opt/splunk/etc/system/metadata/local.meta: Permission denied
Cannot initialize: /opt/splunk/etc/apps/learned/metadata/local.meta: Permission denied
open
ERROR - Failed opening "/opt/splunk/var/log/splunk/splunkd-utility.log": Permission denied
Checking configuration... Error while parsing '/opt/splunk/etc/myinstall/splunkd.xml':
[Errno 13] Permission denied: '/opt/splunk/etc/myinstall/splunkd.xml'

There were problems with the configuration files.
Would you like to ignore these errors? [y/n]: y
Done.
Creating: /opt/splunk/var/run/splunk/appserver/i18n
Warning: cannot create "/opt/splunk/var/run/splunk/appserver/i18n"
[splunk@splunk01 ~]$

Regards Kevin

0 Karma

bosburn_splunk
Splunk Employee
Splunk Employee

A bug has been identified (SPL-87651) where using /etc/init.d/splunk that has been set up by ./splunk enable boot-start -user splunk isn't applying the right GID when accessing files.

This can be worked around as starting Splunk using the user splunk and $SPLUNK_HOME/bin/splunk start vs. /etc/init.d/splunk.

Brian

blebit
Path Finder

Hi,
does it work with other agent versions ?
thanks

0 Karma

BARNEYRUDD
Explorer

I'm on 7.3.1 (universal forwarder) and have this problem. The workaround proposed below by @fharding worked for me.

0 Karma

ccglanville
Engager

Hi Brian,

Do you know when a patch will be released? Is it a simple modification to the /etc/init.d script? I see that the 6.1.x script is missing "/bin/su - splunk -c".

As far as the workaround goes - I assume it won't survive a server reboot?

I also noticed that the files splunk it attempting to read need the splunk user to at least have standard read granted. The ACL access is not enough.

thanks,
Chris

Get Updates on the Splunk Community!

Maximize the Value from Microsoft Defender with Splunk

 Watch NowJoin Splunk and Sens Consulting for this Security Edition Tech TalkWho should attend:  Security ...

This Week's Community Digest - Splunk Community Happenings [6.27.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...

Reminder! Splunk Love Promo: $25 Visa Gift Card for Your Honest SOAR Review With ...

We recently launched our first Splunk Love Special, and it's gone phenomenally well, so we're doing it again, ...