Splunk Connect For Syslog - Sending syslog using Kiwi Syslog Message Generator (UDP 514) failed but TCP 514 success

[azer271]

Hello. I am completely new at Splunk. Recently, I've recently taken on a role where I'll be working with Splunk quite a lot. I have a question about SC4S (Splunk Connect For Syslog). I successfully installed the SC4S (podman + systemd) using the guide from this:

https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/podman-systemd-general/

The SC4S is installed in Centos 7 VM (in vsphere). The HEC is configured successfully in heavy forwarder and I can successfully see the SC4S is properly communicating with Splunk.

After that, I used Kiwi Syslog Message Generator from my windows 10 machine to send a syslog tcp message to the Centos 7 VM.

Successful Output (TCP):

However, if i sent a syslog udp message, the message was not successfully sent. As shown in the screenshot, the messages sent was zero after i pressed send.

Unsuccessful Output (UDP):

No new messages were shown in Splunk Web.

514 TCP and UDP is enabled in the firewall in Centos 7. I would like to request assistance about this issue.

Thank you.

[azer271]

I checked with tcpdump and wireshark. I can clearly see the TCP packets, but not the UDP packets. However, I can see the traffic by echoing the message (TCP and UDP as well) to SC4S server. I believe its the issue of the Kiwi Syslog Message Generator. 

Thanks guys. 😊

[deepakc]

I've never used Kiwi syslog, but you can use the netcat (nc) utility to send test syslog messages to the SC4S server first and check, netcat needs to be installed.  

 UDP test

echo "My Test UDP syslog message" | nc -w1 -u <YOUR SC4S Server> 514

OR locally from the SC4S server

echo "My Test UDP syslog message" | nc -w1 -u localhost 514

And see if any messages are sent to the Splunk/HEC

Also check SC4S to see if data is being sent, when you send data from the Kiwi system 

 sudo tcpdump -i any udp port 514

Other things to check:

 Check the /opt/sc4s/env_file - these are the default ports, but I can't remember if you need to add these as they should be default, may be worth adding these and restarting and see if that could be the cause.

SC4S_LISTEN_DEFAULT_TCP_PORT=514

SC4S_LISTEN_DEFAULT_UDP_PORT=514

Check the logs 

podman logs SC4S

 You said the firewall is ok but might be worth disabling it temporarily.

[PickleRick · Accepted solution]

First and foremost - verify if:

1) The events are generated at the source machine at all - run a wireshark there and see if the packets appear on the wire. If not - here's your culprit - troubleshoot your Kiwi.

2) If they are being sent, check with tcpdump on the receiving end.

3) If you can see the packets on the wire, check firewall rules and rp_filter.