What is the path of Splunk data in any Linux server?

[splunkiri]

I spoke with Linux admin to allow permissions to Splunk app, he asked me what is the path of Splunk logs so that he can allow permissions.. kindly guide !! We can't give root permissions to Splunk forwarder as per policy.

[ephemeric]

IMHO: I think there is some confusion here. The OP wants to ingest logs on the host via the SUF.

So permissions for `splunk` need to be granted on, for example `/var/log/messages` either via group or `setfacl`.

[woodcock]

For forwarders you need read-only for the stuff that you are forwarding and you need write permission for everything under $SPLUNK_HOME which by default on *nix is /opt/splunk/.

[splunkiri]

I already provide the same permissions, kindly have a look,
Still forwarder is not sending the data.. kindly guide..

drwx------ 2 splunk splunk 4096 Oct 16 2016 /opt/splunkforwarder/var/log/introspection
drwx------ 2 splunk splunk 4096 Jul 5 22:02 /opt/splunkforwarder/var/log/splunk

lnx0591:root# ls -ltr
total 261064
-rw------- 1 splunk splunk 0 Oct 16 2016 splunkd_ui_access.log
-rw------- 1 splunk splunk 0 Oct 16 2016 searchhistory.log
-rw------- 1 splunk splunk 0 Oct 16 2016 scheduler.log
-rw------- 1 splunk splunk 0 Oct 16 2016 remote_searches.log
-rw------- 1 splunk splunk 0 Oct 16 2016 mongod.log
-rw------- 1 splunk splunk 0 Oct 16 2016 license_usage.log
-rw------- 1 splunk splunk 0 Oct 16 2016 license_audit.log
-rw------- 1 splunk splunk 64 Oct 16 2016 first_install.log
-rw------- 1 splunk splunk 5817 Jan 9 15:54 splunkd_access.log
-rw------- 1 splunk splunk 25000073 Jan 22 19:58 splunkd.log.5
-rw------- 1 splunk splunk 25000123 Mar 12 09:45 splunkd.log.4
-rw------- 1 splunk splunk 25000040 Apr 29 00:53 splunkd.log.3
-rw------- 1 splunk splunk 299 Jun 14 14:45 splunkd_stdout.log
-rw------- 1 splunk splunk 25000178 Jun 15 10:19 splunkd.log.2
-rw------- 1 splunk splunk 25000171 Jun 26 19:25 metrics.log.5
-rw------- 1 splunk splunk 5825 Jun 28 10:41 splunkd-utility.log
-rw------- 1 splunk splunk 296 Jun 28 10:41 btool.log
-rw------- 1 splunk splunk 482 Jun 28 10:41 splunkd_stderr.log
-rw------- 1 splunk splunk 1336 Jun 28 10:46 conf.log
-rw------- 1 splunk splunk 25000124 Jun 29 02:02 metrics.log.4
-rw------- 1 splunk splunk 25000107 Jun 29 04:23 splunkd.log.1
-rw------- 1 splunk splunk 160573 Jul 1 03:38 audit.log
-rw------- 1 splunk splunk 25000011 Jul 1 08:44 metrics.log.3
-rw------- 1 splunk splunk 25000141 Jul 3 15:25 metrics.log.2
-rw------- 1 splunk splunk 25000088 Jul 5 22:02 metrics.log.1
-rw------- 1 splunk splunk 3887362 Jul 6 06:32 metrics.log
-rw------- 1 splunk splunk 12901867 Jul 6 06:32 splunkd.log
lnx0591:root#

lnx0591:root# ls -lR /opt/splunkforwarder/var/log/
/opt/splunkforwarder/var/log/:
total 8
drwx------ 2 splunk splunk 4096 Oct 16 2016 introspection
drwx------ 2 splunk splunk 4096 Jul 5 22:02 splunk

/opt/splunkforwarder/var/log/introspection:
total 5028
-rw------- 1 splunk splunk 5133404 Jul 6 06:41 disk_objects.log
-rw------- 1 splunk splunk 0 Oct 16 2016 kvstore.log
-rw------- 1 splunk splunk 0 Oct 16 2016 resource_usage.log

/opt/splunkforwarder/var/log/splunk:
total 261140
-rw------- 1 splunk splunk 160573 Jul 1 03:38 audit.log
-rw------- 1 splunk splunk 296 Jun 28 10:41 btool.log
-rw------- 1 splunk splunk 1336 Jun 28 10:46 conf.log
-rw------- 1 splunk splunk 64 Oct 16 2016 first_install.log
-rw------- 1 splunk splunk 0 Oct 16 2016 license_audit.log
-rw------- 1 splunk splunk 0 Oct 16 2016 license_usage.log
-rw------- 1 splunk splunk 3953315 Jul 6 06:41 metrics.log
-rw------- 1 splunk splunk 25000088 Jul 5 22:02 metrics.log.1
-rw------- 1 splunk splunk 25000141 Jul 3 15:25 metrics.log.2
-rw------- 1 splunk splunk 25000011 Jul 1 08:44 metrics.log.3
-rw------- 1 splunk splunk 25000124 Jun 29 02:02 metrics.log.4
-rw------- 1 splunk splunk 25000171 Jun 26 19:25 metrics.log.5
-rw------- 1 splunk splunk 0 Oct 16 2016 mongod.log
-rw------- 1 splunk splunk 0 Oct 16 2016 remote_searches.log
-rw------- 1 splunk splunk 0 Oct 16 2016 scheduler.log
-rw------- 1 splunk splunk 0 Oct 16 2016 searchhistory.log
-rw------- 1 splunk splunk 5817 Jan 9 15:54 splunkd_access.log
-rw------- 1 splunk splunk 12912781 Jul 6 06:40 splunkd.log
-rw------- 1 splunk splunk 25000107 Jun 29 04:23 splunkd.log.1
-rw------- 1 splunk splunk 25000178 Jun 15 10:19 splunkd.log.2
-rw------- 1 splunk splunk 25000040 Apr 29 00:53 splunkd.log.3
-rw------- 1 splunk splunk 25000123 Mar 12 09:45 splunkd.log.4
-rw------- 1 splunk splunk 25000073 Jan 22 19:58 splunkd.log.5
-rw------- 1 splunk splunk 482 Jun 28 10:41 splunkd_stderr.log
-rw------- 1 splunk splunk 299 Jun 14 14:45 splunkd_stdout.log
-rw------- 1 splunk splunk 0 Oct 16 2016 splunkd_ui_access.log
-rw------- 1 splunk splunk 5825 Jun 28 10:41 splunkd-utility.log
lnx0591:root#

[richgalloway]

Have you verified the Forwarder is running as user 'splunk'?

[splunkiri]

Yes, Forwarder is running as a Splunk. I also gave the -rw permissions. but it is sending data only through the sourcetype- Syslog and not from any other. Kindly guide, do we need to give permissions differently to the Splunk user and inside dir.s and files ? and if so, what types of permissions do I need to provide ?

[richgalloway]

I'm at a loss. Are you running SELinux?
Any ideas, @woodcock?

[woodcock]

Yes, SELinux is VERY bad mojo so check that and kill it. Also, what does splunk list monitor show?

[woodcock]

Is this a splunk forwarder or a splunk indexer?

[splunkiri]

Hi, I want to give persmissions to Splunk Forwarder not Splunk indexer. Kindly guide. What permissions do I need to provide ?

[richgalloway]

You're doing well by not running Splunk as root.
Splunk's logs are in $SPLUNK_HOME/var/log/splunk. Permissions should already be granted to the owner of Splunk.

[splunkiri]

Hi, Thanks for the answer. Which permissions should I grant to Splunk directories available in the path $SPLUNK_HOME/var/log/splunk. Kindly reply asap.