Why is Splunk forwarding all logs but ignoring props.conf filter?

[warwickwan]

Greetings!

Sorry, I am a newbie and might be a simple question but I couldn't find any answer works for me.

I'm trying to forward a set of logs to a 3rd party SIEM server using these settings:

outputs.conf

[tcpout]
forwardedindex.4.blacklist = (_internal|_audit|_telemetry|_introspection)

[tcpout:tcp_qradar_172_x_x_x_514]
disabled = false
sendCookedData = false
server = 172.x.x.x:514

props.conf

[google:gcp:pubsub:message]
TRANSFORMS-routing = send_to_qradar_tcp_172_x_x_x_514

transforms.conf

[send_to_qradar_tcp_172_x_x_x_514]
DEST_KEY = _TCP_ROUTING
FORMAT = tcp_qradar_172_x_x_x_514
REGEX = .

However, Splunk forwarded all the logs to the SIEM server, ignoring the filter in props.conf which is "sourcetype=google:gcp:pubsub:message".

Thanks in advance!

[sduff_splunk · Accepted solution]

In your outputs, you may need something like the following

[tcpout]
defaultGroup=normal

[tcpout:normal]
disabled = false
server=10.1.1.1:9997

[tcpout:tcp_qradar_172_x_x_x_514]
disabled = false
sendCookedData = false
server = 172.x.x.x:514

If that doesn't help, can you also provided your inputs.conf as well?

[warwickwan]

Thank you! This does work. Now only the logs matching the filter are forwarded to SIEM. However somehow the other logs are not being indexed locally, and I'm getting the following errors:
10-17-2019 16:30:33.882 -0400 ERROR TcpOutputFd - Read error. Connection reset by peer
10-17-2019 16:30:33.882 -0400 INFO TcpOutputProc - Connection to 172.x.x.3:9997 closed. Read error. Connection reset by peer
10-17-2019 16:30:33.883 -0400 WARN TcpOutputFd - Connect to 172.x.x.3:9997 failed. Connection refused
10-17-2019 16:30:33.883 -0400 ERROR TcpOutputFd - Connection to host=172.x.x.3:9997 failed
10-17-2019 16:30:33.883 -0400 WARN TcpOutputProc - Applying quarantine to ip=172.x.x.3 port=9997 _numberOfFailures=2
10-17-2019 16:30:38.020 -0400 WARN TcpOutputProc - The TCP output processor has paused the data flow. Forwarding to output group normal has been blocked for 11860 seconds. This will probably stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data.

[sduff_splunk]

Can you check that Splunk on 172.x.x.3 is listening on port 9997. Confirm that there are no network issues between the hosts, no firewall or routing issues.