Questions on Splunk and Syslog-ng Server

[splunkfly]

1. What are the Splunk requirements to receive the data from Syslog-ng server?
2. What are the Syslog requirements to get the data from the cisco network devices?
3. What are the Configuration requirements to establish a communication between syslog and cisco devices, and how to configure it?
4. Configuring the separate log files for routers, switches and firewalls.
5. What is the list of the port numbers that listen to Splunk and syslog server?

[mikaelbje]

Q: What are the Configuration requirements to establish a communication between syslog and cisco devices, and how to configure it?

A: Install the Cisco Networks app and add-on, see the Help section in the app.

[mcronkrite]

Q: What are the Splunk requirements to receive the data from Syslog-ng server?
A: Put a Universal Forwarder on your Syslog-NG server and monitor the syslog folder.

Q: What are the Syslog requirements to get the data from the cisco network devices?
A: Configure your cisco devices to send to your syslog device(s)

Q: What are the Configuration requirements to establish a communication between syslog and cisco devices, and how to configure it?
A: Cisco configuration screen has a setting to output to syslog, enter the address of you syslog-ng server, and the delegated port you have selected to listen on. Many people used to choose 514, but that is reserved port, so now days people usually configure 10514 for cisco:asa 20514 for cisco:ios , etc etc

Q: Configuring the separate log files for routers, switches and firewalls.
A:In your cisco log config you set the syslog-ng address and port. You should have a port for each devices type (routers 30514, switches 20514, firewall 10514) This way you can manage the folders and log easily.

Q: What is the list of the port numbers that listen to Splunk and syslog server?
A: The connection between syslog-ng and splunk is the universal fowarder. The universal fowarder outputs to splunk on :9997 by default.
The splunk indexer will listen for universal fowarder on :9997.
The universal forwarder will monitor the log that syslog-ng creates and send the to splunk.

[scottsavaresevi]

ONE. The syslog server should write to a log file. You Splunk forwarder should read that log file in an inputs.conf stanza. Some suggestions here include not having one log file and instead having syslog-ng rotate them every hour (use the day, month, year and hour macros to create different files).

TWO. There aren't many requirements here. But I'll give a few suggestions. Use CNAMEs where possible so that you can move syslog-ng around to different servers if needed and not impact the clients. If that isn't an option, use virtual IPs. Use TCP over UDP as syslog-ng can do multithreading on TCP but can't on UDP (this will help performance and let you maximize how much data syslog-ng can take in). And for security maybe use TLS.

FOUR. You can do this by having the devices go to different syslog-ng ports. So maybe have routers send their syslog to port 514 on the syslog-ng server and firewalls send to port 515 and so on. The alternate is using the same port and either having syslog-ng filter based on the device name (if hostname contains "fw" its a firewall or sw its a switch and so on) but this doesn't scale very well and the more filters you have the slower syslog-ng will perform.

FIVE. I think you are asking what port numbers you can use for this... Any port not used by the OS is fair game. netstat -an on linux will tell you what is currently being used. Spunk won't need ports for this since you are consuming the syslog-ng from a log file written by the syslog-ng process.

[jtacy]

Just curious: do you suggest splitting logs by host or by type only? I currently split by type only because I don't like the idea of remote systems causing the creation of new files on the syslog server (DoS risk?), not to mention the need to keep hundreds or perhaps thousands of open files to support a large environment. I've seen several suggestions to split by host and always wonder if I'm missing something.

[scottsavaresevi]

If you split the logs by host then you may wind up with 100's of files which are hard to manage. Let Splunk properly set the host of the event and you can manage it there. I recommend using the syslog-ng macros to split on the time. Maybe have them rotate every hour. (file-$C_YEAR$C_MONTH$C_DAY$C_HOUR-morestuff). Another suggestion is to have different source types come in on different ports. And each source type gets its own set of logs. This way Splunk doesn't waste time source typing (which is an expensive process and can bog down the indexer).

[jtacy]

Thanks! I've been using the port-based approach for several years and think it works really well. I use a couple of syslog servers behind an F5 VIP and use ports like 10514, 10515, etc. In cases where the source system only supports 514, I route to the correct file using syslog server rules. Thanks for sharing the rationale behind your design!

[clintjd]

Hello. I was trying to route through an F5 VIP, but the syslog-ng would always see the source IP as the VIP, not the originating host. The result was syslog concatenating all of the various host logs into a single file. Would you mind sharing some details about your setup and how you got around this problem? That would be a huge help. Thanks!

[jtacy]

Sure, it sounds like you're using a load balancer that uses source NAT (SNAT) so the server side sees a load balancer IP rather than the client IP. If you're using UDP syslog, you can ask your F5 admins to disable SNAT on your VIP. This only works because syslog servers don't send return traffic back to the client (proper routing of return traffic is a key reason SNAT exists).

Another option, again only if you're using UDP, is to configure a stateless VIP. This is described at https://support.f5.com/csp/article/K13675 and it's great for syslog because each event (packet) can be load balanced individually. This type of VIP doesn't even support SNAT so it will keep your client IPs intact. I haven't run this type of config in production yet but I was impressed by the outcome in testing.

If you're using TCP syslog, you might want to stick with the approach discussed here where syslog listens on multiple ports. Each port would send to a specific file to which you can apply an index and sourcetype in your Splunk config. Unfortunately, you would still have a problem with devices that don't include any sort of host identifier in their events.

Hope this helps, good luck!

[martin_mueller]

1 - Have syslog-ng write log file, install universal forwarder, read log file, forward to indexers.
5 - Ports are fully configurable in Splunk, in your case see inputs.conf