universal forwarder vs. dedicated rsyslog/syslog-ng servers to forward syslog to splunk indexer

[jeff]

Conventional wisdom for collecting syslog data from external sources (network equipment, etc) was to put a couple of dedicated syslog-ng servers behind a load balancer, write the logs to a file, and have Splunk monitor the files. With Splunk 4.2 and the Universal Forwarders, does this still hold true?

We want to add some resiliency in our log collection... in most cases we can use the Splunk Universal Forwarder, but in cases where we can't, we rely on syslog. I was considering deploying a couple of dedicated VMs running Splunk Universal Forwarders behind a load balancer to grab the syslog data from this equipment. I am considering:

1. rsyslog, write to a file, have Splunk Universal Forwarders monitor the files and send to Splunk indexers.
2. Splunk Universal Forwarders listening on port 514/udp, forwarding to Splunk indexer (no dedicated syslog listener)

What are the pros & cons of each approach? My gut tells me that with the proper monitoring and load balancing, the Splunk Universal Forwarder could handle this job by itself.

Thanks.

[BLRINGLER]

Hello

I have a request to have a SYSLOG server and a SPLUNK server. The request is to have the logs from external sources written to the SYSLOG server then forwarded and read by the SPLUNK server.

https://answers.splunk.com/answers/28680/universal-forwarder-vs-dedicated-rsyslog-syslog-ng-servers-...

I am using MS Server 2012 R2 for both, SPLUNK Enterprise 7

How would I:

1. Have logs from different sources (Cisco, Microsoft, Linux) written to a SYSLOG Server.

2. Forward the log to a SPLUNK server

Thanks

[gkanapathy · Accepted solution]

Yes, writing to files (split out by host, with at least one rotated file) is still the recommendation, and the reasons have not changed between 4.1 and 4.2. Three reasons are:

• Performance. For several reasons, you will get better performance on the indexing side, better performance and less resource consumption on the forwarder side, and lower network utilization, if you capture and write to a file with rsyslog instead of capturing UDP packets.
• Reliability. The files provide a buffer of data so that if there are short network or server disruptions, or even extended ones, you don't lose any data.
• Flexibility. If you use rsyslog to split the data by host, you will be able to use [host::] stanzas to do index-time processing on the new (resolved) host names.

[baschny]

I don't see a reason why "Splunk team" cannot implement such a "performant, reliable and flexible" syslog entry point internally so that we don't need that extra stuff in front of it. It's the strenghts of Splunk of being so performant, so why not make a good UDP / syslog compatible entry point for it?

The syslog client implementations can also cache and buffer stuff in case of small network disruptions already.

Papertrail for example can also handle syslog events directly without any problem.

[mwisniewski9]

Is this still the best practice today with Splunk 7

[reedmohn]

As a beginner here, I have to ask about that statement on flexibility: can you give an example of such processing? Or, put in a different way, what is it you can't do, if you go with Splunk UF instead?