Getting Data In

Forward only audit data from /var/log/audit.log

eric_hagen
Explorer

splunk-8.1.0 server RHEL 8 system.

So following the instructions from:

https://docs.splunk.com/Documentation/Splunk/6.6.2/Forwarding/Routeandfilterdatad
Route inputs to specific indexers based on the data input

I configured the following:

vi /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/audit/audit.log]
disabled = false
index = abcd
sourcetype = linux_audit
host = smnloghost
_TCP_ROUTING=monitoring_audit

vi /opt/splunk/etc/system/local/outputs.conf

[tcpout:monitoring_audit]
server = <IP>:<PORT>
type = tcp
disabled = 0

/opt/splunk/bin/splunk cmd btool outputs list tcpout

Shows:

[tcpout:monitoring_audit]
disabled = 0
server = 214.16.207.174:6514
type = tcp

Restarted splunk

No network connection to <IP>, just the rsyslog forwarding syslog data.

netstat -natp | grep <IP>
tcp 0 0 1<IP>:<PORT> <IP>:<PORT> ESTABLISHED 123313/rsyslogd

How can I forward just the data input from the audit log?

 

Labels (2)
Tags (1)
0 Karma
1 Solution

eric_hagen
Explorer

# Documentation was not very clear, below is working

# Do not make changes to system.conf, as it broke a lot of things. Trust in your backup.

# Create cert with encrypted key.

/opt/splunk/etc/auth/dod/system-splunk-smn.pem

-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
-----BEGIN ENCRYPTED PRIVATE KEY-----
<snip>
-----END ENCRYPTED PRIVATE KEY-----

# Verify, it will prompt for password.
openssl s_client -connect <IP>:<PORT> -showcerts \
-cert /opt/splunk/etc/auth/dod/system-splunk-smn.pem \
-CAfile /opt/splunk/etc/auth/dod/system-dod.cacert

# set _TCP_ROUTING=monitoring_audit
vi /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/audit/audit.log]
disabled = false
index = index
sourcetype = linux_audit
host = loghost
_TCP_ROUTING=monitoring_audit

 

#############
# Confiugre output for <IP>:<PORT>
#####
vi /opt/splunk/etc/system/local/outputs.conf

[tcpout]
indexAndForward = true
defaultGroup = monitoring_audit

[tcpout:monitoring_audit]
server = <IP>:<PORT>
disabled = 0
sslRootCAPath = /opt/splunk/etc/auth/dod/system-dod.cacert
sslPassword = <snip>
clientCert = /opt/splunk/etc/auth/dod/system-splunk-smn.pem
useClientSSLCompression = true
sendCookedData = false
sslVerifyServerCert = false
sslVersions = tls1.2

## Test config.
/opt/splunk/bin/splunk cmd btool outputs list tcpout
/opt/splunk/bin/splunk btool outputs list --debug
/opt/splunk/bin/splunk btool check --debug

# Fix any issues

systemctl restart splunk

## Note the sslPassword was setup with plain text, it got encrypted on restart

tcpdump host <IP> -i bond0
tcpdump -vv port <PORT> and "host <IP>" -i bond0

# Verified connection: Queue sometimes grows, the clears.
netstat -natp | grep <PORT>

Verified audit data being indexed into splunk

View solution in original post

0 Karma

eric_hagen
Explorer

# Documentation was not very clear, below is working

# Do not make changes to system.conf, as it broke a lot of things. Trust in your backup.

# Create cert with encrypted key.

/opt/splunk/etc/auth/dod/system-splunk-smn.pem

-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
-----BEGIN ENCRYPTED PRIVATE KEY-----
<snip>
-----END ENCRYPTED PRIVATE KEY-----

# Verify, it will prompt for password.
openssl s_client -connect <IP>:<PORT> -showcerts \
-cert /opt/splunk/etc/auth/dod/system-splunk-smn.pem \
-CAfile /opt/splunk/etc/auth/dod/system-dod.cacert

# set _TCP_ROUTING=monitoring_audit
vi /opt/splunk/etc/apps/search/local/inputs.conf

[monitor:///var/log/audit/audit.log]
disabled = false
index = index
sourcetype = linux_audit
host = loghost
_TCP_ROUTING=monitoring_audit

 

#############
# Confiugre output for <IP>:<PORT>
#####
vi /opt/splunk/etc/system/local/outputs.conf

[tcpout]
indexAndForward = true
defaultGroup = monitoring_audit

[tcpout:monitoring_audit]
server = <IP>:<PORT>
disabled = 0
sslRootCAPath = /opt/splunk/etc/auth/dod/system-dod.cacert
sslPassword = <snip>
clientCert = /opt/splunk/etc/auth/dod/system-splunk-smn.pem
useClientSSLCompression = true
sendCookedData = false
sslVerifyServerCert = false
sslVersions = tls1.2

## Test config.
/opt/splunk/bin/splunk cmd btool outputs list tcpout
/opt/splunk/bin/splunk btool outputs list --debug
/opt/splunk/bin/splunk btool check --debug

# Fix any issues

systemctl restart splunk

## Note the sslPassword was setup with plain text, it got encrypted on restart

tcpdump host <IP> -i bond0
tcpdump -vv port <PORT> and "host <IP>" -i bond0

# Verified connection: Queue sometimes grows, the clears.
netstat -natp | grep <PORT>

Verified audit data being indexed into splunk

0 Karma

eric_hagen
Explorer

telnet to remote host connects.
openssl s_client can connect.

/opt/splunk/var/log/splunk/splunkd.log

11-17-2020 22:24:20.315 +0000 INFO ServerRoles - Declared role=heavyweight_forwarder.
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.0.whitelist , RE : .*
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - found Blacklist forwardedindex.1.blacklist , RE : _.*
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - found Whitelist forwardedindex.2.whitelist , RE : (_audit|_internal|_introspection|_telemetry|_metrics|_metrics_rollup)
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to <IP>:<PORT
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - tcpout group default-autolb-group using Auto load balanced forwarding
11-17-2020 22:24:20.335 +0000 INFO TcpOutputProc - Group default-autolb-group initialized with maxQueueSize=512000 in bytes.
11-17-2020 22:24:20.337 +0000 INFO TcpOutputProc - Initializing connection for non-ssl forwarding to <IP>:PORT
11-17-2020 22:24:20.337 +0000 INFO TcpOutputProc - tcpout group monitoring_audit using Auto load balanced forwarding
11-17-2020 22:24:20.337 +0000 INFO TcpOutputProc - Group monitoring_audit initialized with maxQueueSize=512000 in bytes.
11-17-2020 22:24:20.551 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
11-17-2020 22:24:20.689 +0000 ERROR TcpOutputFd - Read error. Connection reset by peer
11-17-2020 22:24:35.392 +0000 INFO ServerRoles - Undeclared role=heavyweight_forwarder.
11-17-2020 22:24:21.443 +0000 WARN LocalAppsAdminHandler - Using deprecated capabilities for write: admin_all_objects or edit_local_apps. See enable_install_apps in limits.conf
11-17-2020 22:24:31.082 +0000 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=<IP> inside output group default-autolb-group from host_src=host has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
11-17-2020 22:24:35.392 +0000 INFO ServerRoles - Undeclared role=heavyweight_forwarder.
11-17-2020 22:24:41.095 +0000 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to host_dest=<IP> inside output group default-autolb-group from host_src=host has been blocked for blocked_seconds=20. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.
11-17-2020 22:24:41.396 +0000 WARN TcpOutputProc - Send via group=default-autolb-group timed out in 20.026 seconds due to reconfiguration.

 

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...