topic Re: Monitoring/Forwarding data on the Splunk server itself in Getting Data In

Monitoring/Forwarding data on the Splunk server itself

mercierj — Mon, 01 Nov 2021 12:30:50 GMT

My apologies if this question seems mundane or was answered elsewhere but I have searched to no avail. I am completely new to Splunk and am pathfinding the installation and configuration for use as a syslog and audit log store similar to how ELK is often used. While we will add additional data sources at some point my primary focus is on collecting and forwarding /var/log/audit/audit.log and /var/log/auth.log from various Ubuntu hosts into Splunk 8.2(.2.1) Enterprise.

My initial attempt involved installing the UF alongside the Splunk server installation which did not turn out well. Realizing that they are essentially the same daemon and use the same default ports they obviously conflict. So instead I attempted to use the Splunk installation itself as so:

user@splunkhost:~$ sudo /opt/splunk/bin/splunk add forward-server splunkserver:9997 user@splunkhost:~$ sudo /opt/splunk/bin/splunk list forward-server user@splunkhost:~$ sudo /opt/splunk/bin/splunk add monitor /var/log/audit/audit.log -index main -sourcetype %audit-log% user@splunkhost:~$ sudo /opt/splunk/bin/splunk add monitor /var/log/auth.log -index main -sourcetype %auth-log%

However this also did not work and caused the pipeline to essentially become stuck and back up. I believe the error message was something about the TCP output processor pausing the data flow. I am just unsure why.

Essentially I need to collect the security logs from the Splunk server host and index them in Splunk along with everything else but am at a loss as to how this can be accomplished.

Any help or pointers would be most appreciated. Thank you!

Re: Monitoring/Forwarding data on the Splunk server itself

richgalloway — Mon, 01 Nov 2021 17:06:47 GMT

Forwarders are used to convey data to a server running Splunk from those not running Splunk. As you learned, you don't need a separate forwarder on a Splunk server. You also don't need to configure forwarding from Splunk to itself.

Also, it's not recommended to run Splunk as root. Splunk should run as a non-privileged user. Use groups or ACLs to grant read access to the files you wish to monitor.

The splunk add monitor commands should be all that is needed to monitor those files. I have doubts about the sourcetypes in the examples, however, as "%" a character I've never seen used there before.

Re: Monitoring/Forwarding data on the Splunk server itself

mercierj — Mon, 01 Nov 2021 18:58:15 GMT

Thanks. I am actually not running Splunk as root (it runs as Splunk) but I can't log in as that user. I could probably get around it using sudo /usr/sbin/runuser -s /bin/bash splunk so I can run those commands as the proper user. Perhaps running those as root created files Splunk couldn't read. I'll give it a try, report back, and give you the upvote once I can verify. Thanks again!

Re: Monitoring/Forwarding data on the Splunk server itself

isoutamo — Mon, 01 Nov 2021 19:11:01 GMT

Does sudo -u splunk bash or sudo-u splunk /opt/splunk/bin/splunk xxxx work? Those are usually the way how you should run commands when user splunk hasn’t login shell/rights.

Re: Monitoring/Forwarding data on the Splunk server itself

mercierj — Mon, 01 Nov 2021 19:18:15 GMT

You would think, but our sudo maps are centrally stored in LDAP and heavily locked down for all manner of shell escapes and subshelling. I'd blame the admin but I am one of them!

Re: Monitoring/Forwarding data on the Splunk server itself

richgalloway — Tue, 02 Nov 2021 19:11:17 GMT

Yes, running Splunk CLI commands as root when Splunk itself is not running as root runs the risk of files being written that cannot be read by user splunk. Or maybe the file can be read by splunk but can't be updated. Best to avoid that and run CLI commands as the proper user. Most places I've been doing that by using sudo to switch users. Something like sudo -iu splunk or sudo su splunk. Talk to the powers that be and strongly encourage them to allow one of these commands for the health and safety of Splunk.

Re: Monitoring/Forwarding data on the Splunk server itself

isoutamo — Mon, 01 Nov 2021 19:43:38 GMT

Another option until you could fix sudoers is sudo chown -R splunk:splunk /opt/splunk/ after you have added those and probably restart splunk?

Re: Monitoring/Forwarding data on the Splunk server itself

mercierj — Tue, 02 Nov 2021 18:24:32 GMT

Thanks, I tried this and it worked. For those who are curious as to what I ended up doing on the server to get it running as 'splunk' with systemd and monitoring own files:

user@splunkserver:~$ sudo apt install splunk user@splunkserver:~$ sudo /opt/splunk/bin/splunk start --accept-license This appears to be your first time running this version of Splunk. ... user@splunkhost:~$ sudo gpasswd -a splunk adm # let splunk read the logs user@splunkhost:~$ sudo /opt/splunk/bin/splunk stop user@splunkhost:~$ sudo chown -R splunk:splunk /opt/splunk user@splunkhost:~$ sudo /opt/splunk/bin/splunk enable boot-start -user splunk -systemd-managed 1 user@splunkhost:~$ sudo systemctl enable --now Splunkd user@splunkhost:~$ sudo -u splunk /opt/splunk/bin/splunk enable listen 9997 user@splunkhost:~$ sudo -u splunk /opt/splunk/bin/splunk add monitor /var/log/audit/audit.log -index main -sourcetype audit-log user@splunkhost:~$ sudo -u splunk /opt/splunk/bin/splunk add monitor /var/log/auth.log -index main -sourcetype auth-log

On the other nodes I installed the universal forwarder as normal (which runs as 'splunk' by default):

user@client:~$ sudo apt install splunkforwarder user@client:~$ sudo /opt/splunkforwarder/bin/splunk enable boot-start -systemd-managed 1 --accept-license This appears to be your first time running this version of Splunk. user@client:~$ sudo gpasswd -a splunk adm # allow splunk to read the logs user@client:~$ sudo systemctl enable --now SplunkForwarder user@client:~$ sudo -u splunk /opt/splunkforwarder/bin/splunk add forward-server splunkserver:9997 user@client:~$ sudo -u splunk /opt/splunkforwarder/bin/splunk list forward-server user@client:~$ sudo -u splunk /opt/splunkforwarder/bin/splunk add monitor /var/log/audit/audit.log -index main -sourcetype audit-log user@client:~$ sudo -u splunk /opt/splunkforwarder/bin/splunk add monitor /var/log/auth.log -index main -sourcetype auth-log

As for the % sigil in the source labels I got that from this link but I have since removed them:

Solved: How do I configure a Splunk Forwarder on Linux? - Splunk Community

I may look into using the Add-on For UNIX and Linux but honestly this was pretty simple and meets my immediate needs. Thanks for the help!