Getting Data In

Splunk Connect For Syslog - Sending syslog using Kiwi Syslog Message Generator (UDP 514) failed but TCP 514 success

azer271
Explorer

Hello. I am completely new at Splunk. Recently, I've recently taken on a role where I'll be working with Splunk quite a lot. I have a question about SC4S (Splunk Connect For Syslog). I successfully installed the SC4S (podman + systemd) using the guide from this:

https://splunk.github.io/splunk-connect-for-syslog/main/gettingstarted/podman-systemd-general/

The SC4S is installed in Centos 7 VM (in vsphere). The HEC is configured successfully in heavy forwarder and I can successfully see the SC4S is properly communicating with Splunk.

azer271_0-1715008046922.png

After that, I used Kiwi Syslog Message Generator from my windows 10 machine to send a syslog tcp message to the Centos 7 VM.

azer271_1-1715008106499.png

Successful Output (TCP):

azer271_2-1715008125518.png

However, if i sent a syslog udp message, the message was not successfully sent. As shown in the screenshot, the messages sent was zero after i pressed send.

azer271_3-1715008305699.png

Unsuccessful Output (UDP):

azer271_4-1715008356739.png

No new messages were shown in Splunk Web.

514 TCP and UDP is enabled in the firewall in Centos 7. I would like to request assistance about this issue.

Thank you.

 

 

 

 

 
 
 
 
 
Labels (6)
0 Karma
1 Solution

PickleRick
SplunkTrust
SplunkTrust

First and foremost - verify if:

1) The events are generated at the source machine at all - run a wireshark there and see if the packets appear on the wire. If not - here's your culprit - troubleshoot your Kiwi.

2) If they are being sent, check with tcpdump on the receiving end.

3) If you can see the packets on the wire, check firewall rules and rp_filter.

 

View solution in original post

azer271
Explorer

I checked with tcpdump and wireshark. I can clearly see the TCP packets, but not the UDP packets. However, I can see the traffic by echoing the message (TCP and UDP as well) to SC4S server. I believe its the issue of the Kiwi Syslog Message Generator. 

Thanks guys. 😊

0 Karma

deepakc
Builder

I've never used Kiwi syslog, but you can use the netcat (nc) utility to send test syslog messages to the SC4S server first and check, netcat needs to be installed.  

 UDP test

echo "My Test UDP syslog message" | nc -w1 -u <YOUR SC4S Server> 514

OR locally from the SC4S server

echo "My Test UDP syslog message" | nc -w1 -u localhost 514

And see if any messages are sent to the Splunk/HEC

Also check SC4S to see if data is being sent, when you send data from the Kiwi system 

 sudo tcpdump -i any udp port 514

 

Other things to check:

 Check the /opt/sc4s/env_file - these are the default ports, but I can't remember if you need to add these as they should be default, may be worth adding these and restarting and see if that could be the cause.

 

SC4S_LISTEN_DEFAULT_TCP_PORT=514

SC4S_LISTEN_DEFAULT_UDP_PORT=514

 

Check the logs 

podman logs SC4S

 You said the firewall is ok but might be worth disabling it temporarily.

PickleRick
SplunkTrust
SplunkTrust

First and foremost - verify if:

1) The events are generated at the source machine at all - run a wireshark there and see if the packets appear on the wire. If not - here's your culprit - troubleshoot your Kiwi.

2) If they are being sent, check with tcpdump on the receiving end.

3) If you can see the packets on the wire, check firewall rules and rp_filter.

 

Get Updates on the Splunk Community!

Splunk is Nurturing Tomorrow’s Cybersecurity Leaders Today

Meet Carol Wright. She leads the Splunk Academic Alliance program at Splunk. The Splunk Academic Alliance ...

Part 2: A Guide to Maximizing Splunk IT Service Intelligence

Welcome to the second segment of our guide. In Part 1, we covered the essentials of getting started with ITSI ...

Part 1: A Guide to Maximizing Splunk IT Service Intelligence

As modern IT environments continue to grow in complexity and speed, the ability to efficiently manage and ...