Forward only audit data from /var/log/audit.log

[eric_hagen]

splunk-8.1.0 server RHEL 8 system.

So following the instructions from:

https://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad
Route inputs to specific indexers based on the data input

I configured the following:

vi /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/audit/audit.log]
disabled = false
index = abcd
sourcetype = linux_audit
host = smnloghost
_TCP_ROUTING=monitoring_audit

vi /opt/splunk/etc/system/local/outputs.conf

[tcpout:monitoring_audit]
server = <IP>:<PORT>
type = tcp
disabled = 0

/opt/splunk/bin/splunk cmd btool outputs list tcpout

Shows:

[tcpout:monitoring_audit]
disabled = 0
server = 214.16.207.174:6514
type = tcp

Restarted splunk

No network connection to <IP>, just the rsyslog forwarding syslog data.

netstat -natp | grep <IP>
tcp 0 0 1<IP>:<PORT> <IP>:<PORT> ESTABLISHED 123313/rsyslogd

How can I forward just the data input from the audit log?

[eric_hagen · Accepted solution]

# Documentation was not very clear, below is working

# Do not make changes to system.conf, as it broke a lot of things. Trust in your backup.

# Create cert with encrypted key.

/opt/splunk/etc/auth/dod/system-splunk-smn.pem

-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
-----BEGIN ENCRYPTED PRIVATE KEY-----
<snip>
-----END ENCRYPTED PRIVATE KEY-----

# Verify, it will prompt for password.
openssl s_client -connect <IP>:<PORT> -showcerts \
-cert /opt/splunk/etc/auth/dod/system-splunk-smn.pem \
-CAfile /opt/splunk/etc/auth/dod/system-dod.cacert

# set _TCP_ROUTING=monitoring_audit
vi /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/audit/audit.log]
disabled = false
index = index
sourcetype = linux_audit
host = loghost
_TCP_ROUTING=monitoring_audit

#############
# Confiugre output for <IP>:<PORT>
#####
vi /opt/splunk/etc/system/local/outputs.conf

[tcpout]
indexAndForward = true
defaultGroup = monitoring_audit

[tcpout:monitoring_audit]
server = <IP>:<PORT>
disabled = 0
sslRootCAPath = /opt/splunk/etc/auth/dod/system-dod.cacert
sslPassword = <snip>
clientCert = /opt/splunk/etc/auth/dod/system-splunk-smn.pem
useClientSSLCompression = true
sendCookedData = false
sslVerifyServerCert = false
sslVersions = tls1.2

## Test config.
/opt/splunk/bin/splunk cmd btool outputs list tcpout
/opt/splunk/bin/splunk btool outputs list --debug
/opt/splunk/bin/splunk btool check --debug

# Fix any issues

systemctl restart splunk

## Note the sslPassword was setup with plain text, it got encrypted on restart

tcpdump host <IP> -i bond0
tcpdump -vv port <PORT> and "host <IP>" -i bond0

# Verified connection: Queue sometimes grows, the clears.
netstat -natp | grep <PORT>

Verified audit data being indexed into splunk

[eric_hagen]

telnet to remote host connects.
openssl s_client can connect.

/opt/splunk/var/log/splunk/splunkd.log

11-17-2020 22:24:20.315 +0000 INFO ServerRoles - Declared role=heavyweight_forwarder.
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .*
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.*
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to <IP>:<PORT
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - tcpout group default-autolb-group using Auto load balanced forwarding
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - Group default-autolb-group initialized with maxQueueSize=512000 in bytes.
11-17-2020 22:24:20.337 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to <IP>:PORT
11-17-2020 22:24:20.337 +0000 INFO TcpOutputProc - tcpout group monitoring_audit using Auto load balanced forwarding
11-17-2020 22:24:20.337 +0000 INFO TcpOutputProc - Group monitoring_audit initialized with maxQueueSize=512000 in bytes.
11-17-2020 22:24:20.551 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
11-17-2020 22:24:20.689 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
11-17-2020 22:24:35.392 +0000 INFO ServerRoles - Undeclared role=heavyweight_forwarder.
11-17-2020 22:24:21.443 +0000 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
11-17-2020 22:24:31.082 +0000 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=<IP> inside output group default-autolb-group from host_src=host has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
11-17-2020 22:24:35.392 +0000 INFO ServerRoles - Undeclared role=heavyweight_forwarder.
11-17-2020 22:24:41.095 +0000 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=<IP> inside output group default-autolb-group from host_src=host has been blocked for blocked_seconds=20. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
11-17-2020 22:24:41.396 +0000 WARN TcpOutputProc - Send via group=default-autolb-group timed out in 20.026 seconds due to reconfiguration.