Getting Data In

Forward only audit data from /var/log/audit.log

eric_hagen
Explorer

splunk-8.1.0 server RHEL 8 system.

So following the instructions from:

https://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad
Route inputs to specific indexers based on the data input

I configured the following:

vi /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/audit/audit.log]
disabled = false
index = abcd
sourcetype = linux_audit
host = smnloghost
_TCP_ROUTING=monitoring_audit

vi /opt/splunk/etc/system/local/outputs.conf

[tcpout:monitoring_audit]
server = <IP>:<PORT>
type = tcp
disabled = 0

/opt/splunk/bin/splunk cmd btool outputs list tcpout

Shows:

[tcpout:monitoring_audit]
disabled = 0
server = 214.16.207.174:6514
type = tcp

Restarted splunk

No network connection to <IP>, just the rsyslog forwarding syslog data.

netstat -natp | grep <IP>
tcp 0 0 1<IP>:<PORT> <IP>:<PORT> ESTABLISHED 123313/rsyslogd

How can I forward just the data input from the audit log?

 

Labels (2)
Tags (1)
0 Karma
1 Solution

eric_hagen
Explorer

# Documentation was not very clear, below is working

# Do not make changes to system.conf, as it broke a lot of things. Trust in your backup.

# Create cert with encrypted key.

/opt/splunk/etc/auth/dod/system-splunk-smn.pem

-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
-----BEGIN ENCRYPTED PRIVATE KEY-----
<snip>
-----END ENCRYPTED PRIVATE KEY-----

# Verify, it will prompt for password.
openssl s_client -connect <IP>:<PORT> -showcerts \
-cert /opt/splunk/etc/auth/dod/system-splunk-smn.pem \
-CAfile /opt/splunk/etc/auth/dod/system-dod.cacert

# set _TCP_ROUTING=monitoring_audit
vi /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/audit/audit.log]
disabled = false
index = index
sourcetype = linux_audit
host = loghost
_TCP_ROUTING=monitoring_audit

 

#############
# Confiugre output for <IP>:<PORT>
#####
vi /opt/splunk/etc/system/local/outputs.conf

[tcpout]
indexAndForward = true
defaultGroup = monitoring_audit

[tcpout:monitoring_audit]
server = <IP>:<PORT>
disabled = 0
sslRootCAPath = /opt/splunk/etc/auth/dod/system-dod.cacert
sslPassword = <snip>
clientCert = /opt/splunk/etc/auth/dod/system-splunk-smn.pem
useClientSSLCompression = true
sendCookedData = false
sslVerifyServerCert = false
sslVersions = tls1.2

## Test config.
/opt/splunk/bin/splunk cmd btool outputs list tcpout
/opt/splunk/bin/splunk btool outputs list --debug
/opt/splunk/bin/splunk btool check --debug

# Fix any issues

systemctl restart splunk

## Note the sslPassword was setup with plain text, it got encrypted on restart

tcpdump host <IP> -i bond0
tcpdump -vv port <PORT> and "host <IP>" -i bond0

# Verified connection: Queue sometimes grows, the clears.
netstat -natp | grep <PORT>

Verified audit data being indexed into splunk

View solution in original post

0 Karma

eric_hagen
Explorer

# Documentation was not very clear, below is working

# Do not make changes to system.conf, as it broke a lot of things. Trust in your backup.

# Create cert with encrypted key.

/opt/splunk/etc/auth/dod/system-splunk-smn.pem

-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
-----BEGIN ENCRYPTED PRIVATE KEY-----
<snip>
-----END ENCRYPTED PRIVATE KEY-----

# Verify, it will prompt for password.
openssl s_client -connect <IP>:<PORT> -showcerts \
-cert /opt/splunk/etc/auth/dod/system-splunk-smn.pem \
-CAfile /opt/splunk/etc/auth/dod/system-dod.cacert

# set _TCP_ROUTING=monitoring_audit
vi /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/audit/audit.log]
disabled = false
index = index
sourcetype = linux_audit
host = loghost
_TCP_ROUTING=monitoring_audit

 

#############
# Confiugre output for <IP>:<PORT>
#####
vi /opt/splunk/etc/system/local/outputs.conf

[tcpout]
indexAndForward = true
defaultGroup = monitoring_audit

[tcpout:monitoring_audit]
server = <IP>:<PORT>
disabled = 0
sslRootCAPath = /opt/splunk/etc/auth/dod/system-dod.cacert
sslPassword = <snip>
clientCert = /opt/splunk/etc/auth/dod/system-splunk-smn.pem
useClientSSLCompression = true
sendCookedData = false
sslVerifyServerCert = false
sslVersions = tls1.2

## Test config.
/opt/splunk/bin/splunk cmd btool outputs list tcpout
/opt/splunk/bin/splunk btool outputs list --debug
/opt/splunk/bin/splunk btool check --debug

# Fix any issues

systemctl restart splunk

## Note the sslPassword was setup with plain text, it got encrypted on restart

tcpdump host <IP> -i bond0
tcpdump -vv port <PORT> and "host <IP>" -i bond0

# Verified connection: Queue sometimes grows, the clears.
netstat -natp | grep <PORT>

Verified audit data being indexed into splunk

0 Karma

eric_hagen
Explorer

telnet to remote host connects.
openssl s_client can connect.

/opt/splunk/var/log/splunk/splunkd.log

11-17-2020 22:24:20.315 +0000 INFO ServerRoles - Declared role=heavyweight_forwarder.
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .*
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.*
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to <IP>:<PORT
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - tcpout group default-autolb-group using Auto load balanced forwarding
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - Group default-autolb-group initialized with maxQueueSize=512000 in bytes.
11-17-2020 22:24:20.337 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to <IP>:PORT
11-17-2020 22:24:20.337 +0000 INFO TcpOutputProc - tcpout group monitoring_audit using Auto load balanced forwarding
11-17-2020 22:24:20.337 +0000 INFO TcpOutputProc - Group monitoring_audit initialized with maxQueueSize=512000 in bytes.
11-17-2020 22:24:20.551 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
11-17-2020 22:24:20.689 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
11-17-2020 22:24:35.392 +0000 INFO ServerRoles - Undeclared role=heavyweight_forwarder.
11-17-2020 22:24:21.443 +0000 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
11-17-2020 22:24:31.082 +0000 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=<IP> inside output group default-autolb-group from host_src=host has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
11-17-2020 22:24:35.392 +0000 INFO ServerRoles - Undeclared role=heavyweight_forwarder.
11-17-2020 22:24:41.095 +0000 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=<IP> inside output group default-autolb-group from host_src=host has been blocked for blocked_seconds=20. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
11-17-2020 22:24:41.396 +0000 WARN TcpOutputProc - Send via group=default-autolb-group timed out in 20.026 seconds due to reconfiguration.

 

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...