Running rsyslog and Splunk Enterprise on the same machine

[jkamdar]

Hi, I have a small lab (air gapped) with about 2 Linux servers  not including the Splunk server and 25 Windows machine.

I have deployed Splunk and ingesting logs from all Linux and Windows clients and also from network switch, VMWare server and hosts.

I am able to send logs from network switch and VMWare hosts directly into Splunk using using "Data Inputs->TCP" and by picking different ports for each service but for Cisco UCS Chassis, to send logs, I can't configure other than syslog server name and log level. 

So I setup a rsyslog server on the same machine as Splunk Enterprise. It seems to be running but I don't logs from Cisco UCS. I have check firewall rules as well and all seems to be configured properly. 

Any tips about running rsyslog and Splunk server on the same machine and about sending Cisco UCS logs to rsyslog/splunk would be appreciated. 

Unfortunately, I can't provide much info as this is an air gapped lab. 

[jkamdar · Accepted solution]

Just wanted to report, the problem has been solved. Everything remained same, I just restarted rsyslog and I started seeing logs on the rsyslog server; when in doubt, reboot seemed to have worked here 🙂

[jkamdar]

@PickleRick thanks for your response. Yes, it's configured properly but tcpdump showed nothing coming to port 514. It seems the problem might be on the UCS side. As someone on the Cisco community suggested, tried to run on UCS side "ethanalyzer local interface mgmt capture-filter "port 514" limit-captured-frames 0 detail" but looks like it's not generating any traffic to send out port 514 on UCS itself and hence no data on the rsyso

[PickleRick]

There is absolutely no problem with running rsyslog on the same box as splunk provided that you're not trying to bind the same port(s) to both programs.

Have you configured rsyslog to receive network data on proper ports? Did you verify it is listening? Did you check with tcpdump/wireshark whether UCS is sending data?