Sending Syslog to a Third Party Is Not Working

[phil_wang]

I have configured forwarding syslog to a third party device but seems the Splunk Heavy Forwarder is not forwarding the syslog. Could someone take a look at this and point out the obvious error or to a right direction please?

1. I can see syslog stream is coming in on UDP 514 from IP x.x.33.115 and the syslogs are appearing on the Splunk HF metric.log

psplunkls03:~ # more /data1/splunk/var/log/splunk/metrics.log | grep x.x.32.115
07-18-2016 16:09:44.426 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=282.791899, eps=1685.059326, kb=14139.600586, ev=84253, avg_age=6.434192, max_age=16
07-18-2016 16:09:44.427 +1000 INFO Metrics - group=udpin_connections, x.x.32.115:514, sourcePort=514, _udp_bps=226718.54, _udp_kbps=221.40, _udp_avg_thruput=228.32, _udp_kprocessed=452005.39, _udp_eps=1552.66
07-18-2016 16:10:32.426 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=302.518184, eps=1797.906808, kb=14520.784180, ev=86299, avg_age=6.887612, max_age=15
07-18-2016 16:10:32.426 +1000 INFO Metrics - group=udpin_connections, x.x.32.115:514, sourcePort=514, _udp_bps=262810.17, _udp_kbps=256.65, _udp_avg_thruput=228.84, _udp_kprocessed=466749.14, _udp_eps=1795.05
07-18-2016 16:11:24.425 +1000 INFO Metrics - group=per_host_thruput, series="x.x.32.115", kbps=209.181001, eps=1246.029630, kb=10877.321289, ev=64793, avg_age=10.525211, max_age=20

1. Configuration on Splunk Heavy Forwarder:

psplunkls03:~ # more /data1/splunk/etc/apps/search/local/inputs.conf
[splunktcp://9997]
connection_host = ip

[udp://x.x.32.115:514]
connection_host = ip
index = index_asa
sourcetype = cisco:asa
disabled = 0
…..

psplunkls03:~ # more /data1/splunk/etc/system/local/outputs.conf
[syslog]
defaultGroup=syslogGroup

[syslog:my_syslog_group]
server = 192.168.16.194:514

psplunkls03:~ # more /data1/splunk/etc/system/local/transform.conf
[send_to_mssp]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = my_syslog_group

psplunkls03:~ # more /data1/splunk/etc/system/local/props.conf
[host::x.x.32.115]
TRANSFORMS-asa = send_to_mssp

1. I can see Splunk Heavy Forwarder is forwarding empty packets to the designated IP 192.168.16.194 on TCP 514 though, which itself is a bit strange as it should be UDP as well right?

psplunkls03:~ # more /data1/splunk/var/log/splunk/metrics.log | grep 192.168.16.194
07-18-2016 16:32:16.341 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:33:06.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:33:58.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:34:50.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:35:28.338 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00
07-18-2016 16:36:00.336 +1000 INFO Metrics - group=syslog_connections, my_syslog_group:192.168.16.194:514:192.168.16.194:514, sourcePort=8089, destIp=192.168.16.194, destPort=514, _tcp_Bps=0.00, _tcp_KBps=0.00, _tcp_avg_thruput=0.00, _tcp_Kprocessed=0, _tcp_eps=0.00

[hmclaren_splunk]

Just a thought, but is there any reason you're not using a syslog collector (rsyslog / syslog-ng) then having Splunk read the log files? This would be the prefered option, you could also then route the syslog stream to both the local file system and to the external third party system you're aiming for.

You could try looking through this answer which details a successful config for third party routing via Splunk Heavy Forwarder: https://answers.splunk.com/answers/65818/forward-data-to-a-third-party-system.html

You're right that the data should default to UDP and NOT TCP.

Do the quantity of 'empty' forwarded packets match with the received syslog data?

On your props.conf, is [host::x.x.32.115]are the 'x's accurate regex? Should they be '*' with escaped '.'?
I assume you have checked your host field is an IP address?