Monitoring/Forwarding data on the Splunk server itself

[mercierj]

My apologies if this question seems mundane or was answered elsewhere but I have searched to no avail.  I am completely new to Splunk and am pathfinding the installation and configuration for use as a syslog and audit log store similar to how ELK is often used.  While we will add additional data sources at some point my primary focus is on collecting and forwarding /var/log/audit/audit.log and /var/log/auth.log from various Ubuntu hosts into Splunk 8.2(.2.1) Enterprise.

My initial attempt involved installing the UF alongside the Splunk server installation which did not turn out well.  Realizing that they are essentially the same daemon and use the same default ports they obviously conflict.  So instead I attempted to use the Splunk installation itself as so:

user@splunkhost:~$ sudo /opt/splunk/bin/splunk add forward-server splunkserver:9997
user@splunkhost:~$ sudo /opt/splunk/bin/splunk list forward-server
user@splunkhost:~$ sudo /opt/splunk/bin/splunk add monitor /var/log/audit/audit.log -index main -sourcetype %audit-log%
user@splunkhost:~$ sudo /opt/splunk/bin/splunk add monitor /var/log/auth.log -index main -sourcetype %auth-log%

 However this also did not work and caused the pipeline to essentially become stuck and back up.  I believe the error message was something about the TCP output processor pausing the data flow.  I am just unsure why.

Essentially I need to collect the security logs from the Splunk server host and index them in Splunk along with everything else but am at a loss as to how this can be accomplished.

Any help or pointers would be most appreciated.  Thank you!

[richgalloway · Accepted solution]

Forwarders are used to convey data to a server running Splunk from those not running Splunk.  As you learned, you don't need a separate forwarder on a Splunk server.  You also don't need to configure forwarding from Splunk to itself.

Also, it's not recommended to run Splunk as root.  Splunk should run as a non-privileged user.  Use groups or ACLs to grant read access to the files you wish to monitor.

The splunk add monitor commands should be all that is needed to monitor those files.  I have doubts about the sourcetypes in the examples, however, as "%" a character I've never seen used there before.

[mercierj]

Thanks, I tried this and it worked.  For those who are curious as to what I ended up doing on the server to get it running as 'splunk' with systemd and monitoring own files:

user@splunkserver:~$ sudo apt install splunk
user@splunkserver:~$ sudo /opt/splunk/bin/splunk start --accept-license
This appears to be your first time running this version of Splunk.
...
user@splunkhost:~$ sudo gpasswd -a splunk adm # let splunk read the logs
user@splunkhost:~$ sudo /opt/splunk/bin/splunk stop
user@splunkhost:~$ sudo chown -R splunk:splunk /opt/splunk
user@splunkhost:~$ sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 1
user@splunkhost:~$ sudo systemctl enable --now Splunkd
user@splunkhost:~$ sudo -u splunk /opt/splunk/bin/splunk enable listen 9997
user@splunkhost:~$ sudo -u splunk /opt/splunk/bin/splunk add monitor /var/log/audit/audit.log -index main -sourcetype audit-log
user@splunkhost:~$ sudo -u splunk /opt/splunk/bin/splunk add monitor /var/log/auth.log -index main -sourcetype auth-log

On the other nodes I installed the universal forwarder as normal (which runs as 'splunk' by default):

user@client:~$ sudo apt install splunkforwarder
user@client:~$ sudo /opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 1 --accept-license
This appears to be your first time running this version of Splunk.
user@client:~$ sudo gpasswd -a splunk adm # allow splunk to read the logs
user@client:~$ sudo systemctl enable --now SplunkForwarder
user@client:~$ sudo -u splunk /opt/splunkforwarder/bin/splunk add forward-server splunkserver:9997
user@client:~$ sudo -u splunk /opt/splunkforwarder/bin/splunk list forward-server
user@client:~$ sudo -u splunk /opt/splunkforwarder/bin/splunk add monitor /var/log/audit/audit.log -index main -sourcetype audit-log
user@client:~$ sudo -u splunk /opt/splunkforwarder/bin/splunk add monitor /var/log/auth.log -index main -sourcetype auth-log

 As for the % sigil in the source labels I got that from this link but I have since removed them:

 Solved: How do I configure a Splunk Forwarder on Linux? - Splunk Community

I may look into using the Add-on For UNIX and Linux but honestly this was pretty simple and meets my immediate needs.  Thanks for the help!

[mercierj]

Thanks.  I am actually not running Splunk as root (it runs as Splunk) but I can't log in as that user.  I could probably get around it using sudo /usr/sbin/runuser -s /bin/bash splunk so I can run those commands as the proper user.  Perhaps running those as root created files Splunk couldn't read.  I'll give it a try, report back, and give you the upvote once I can verify.  Thanks again!

[richgalloway]

Yes, running Splunk CLI commands as root when Splunk itself is not running as root runs the risk of files being written that cannot be read by user splunk.  Or maybe the file can be read by splunk but can't be updated.  Best to avoid that and run CLI commands as the proper user.  Most places I've been doing that by using sudo to switch users.  Something like sudo -iu splunk or sudo su splunk.  Talk to the powers that be and strongly encourage them to allow one of these commands for the health and safety of Splunk.

[isoutamo]

Another option until you could fix sudoers is sudo chown -R splunk:splunk /opt/splunk/ after you have added those and probably restart splunk?

[isoutamo]

Does sudo -u splunk bash or sudo-u splunk /opt/splunk/bin/splunk xxxx work? Those are usually the way how you should run commands when user splunk hasn’t login shell/rights.

[mercierj]

You would think, but our sudo maps are centrally stored in LDAP and heavily locked down for all manner of shell escapes and subshelling.  I'd blame the admin but I am one of them!