topic Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk? in Getting Data In

Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

dglinder — Mon, 23 Sep 2013 02:31:44 GMT

For my installation (that I've inherited from multiple administrators), we have some events coming in through direct TCP/UDP 514 syslog events to Splunk, and others come in through a log file that Syslog-NG creates and updates. Can anyone add additional thoughts/comments to my pros/cons list below?

Benefits of Syslog-NG over Splunk for syslog reception:

Pros

Separate process from Splunk
Buffer incoming events if/when Splunk process not available
Syslog-NG config can be updated without restarting Syslog-NG process (SIGHUP - re-read config files)
kristian.kolb: The syslog server can structure the incoming logs into a directory structure on the file system, based on who is sending. This makes it easier to set up proper host, source, and sourcetype configuration with ordinary [monitor] stanzas, which in turn simplifies field extraction etc.

Cons

One more service/program to update and monitor
Possible (?) delay between receipt of syslog message and appearance of message in Splunk.

To me, the biggest benefits are the ability to restart my Splunk processes without loosing incoming events. But, since the restart is quite rare I'm concerned that the delay mentioned as the second con might be substantial (double-digit minutes or longer).

Is this delay concern warranted?

Are there additional cons that I overlooked?

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

czanik — Mon, 23 Sep 2013 09:02:38 GMT

From the syslog-ng side, we put together the following doc: http://www.balabit.com/support/documentation/pdf/syslog-ng_splunk_deployment_guide_en.pdf

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

kristian_kolb — Mon, 23 Sep 2013 10:27:28 GMT

Why would you run into double-digit minute delays? That sounds pretty awful. Normally you'll have a delay that could be measured in single-digit seconds, in my experience.

In any case, I think you have the pro's listed correctly. One more addition is that the syslog server can structure the incoming logs into a directory structure on the file system, based on who is sending. This makes it easier to set up proper host, source, and sourcetype configuration with ordinary [monitor] stanzas, which in turn simplifies field extraction etc.

The alternative would be to have everything coming in to Splunk on a single port, and then try to structure it from there, with index-time transformations of sourcetypes etc. Or to set up syslog sending over several ports, e.g. a separate port for each host sending logs.. brrr .

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

Narj — Mon, 23 Sep 2013 11:02:21 GMT

We're using syslog-ng and are very happy with it... we already had central syslog receivers before splunk, so moving to syslog-ng has enabled us to split logs out nicely for the syslog light forwarder to pick up and stream to the indexer. This meant we could get splunk in with minimum effort (not having to reconfigure all network devices, update firewalls etc etc).

Splitting your logs out in such a fashion also means that you can use a blacklist/whitelist in Splunk in case the need arises to manage occasional issues with log volume exceeding the licence. There's also the fact that you can still access your logs (albeit more painfully) if Splunk dies for whatever reason.

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

dglinder — Mon, 23 Sep 2013 13:16:45 GMT

Regarding "double-digit minute delays", I was purely speculating on a worst-case scenario. I can't imagine that case either, except possibly on a heavily loaded syslog-ng server.

One of the inherited designs is just as you described in the third paragraph (Splunk listening on several ports). The complications it adds is one of the main reasons I brought up this pro/con discussion. Thanks!

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

dglinder — Mon, 23 Sep 2013 13:24:56 GMT

Thanks, I'll check it out.

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

yannK — Mon, 23 Sep 2013 18:00:28 GMT

pro:

the syslog to file acts as a file buffer, so much better than relying on tcp queue and memory.
You do not loose any events when splunk is down or restarting.

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

dglinder — Mon, 23 Sep 2013 18:03:59 GMT

Agreed, thanks!

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

Narj — Tue, 24 Sep 2013 12:41:03 GMT

How typical is this... sing syslog-ng's praises and it stops working overnight after log rotation! O_o First problem in years!

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

dglinder — Tue, 24 Sep 2013 13:27:23 GMT

I hear your pain! Though by separating the Splunk duties from the Syslog duties, the main function of one was not impacted during the outage/restart of the other.

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

vladx — Tue, 31 Jul 2018 11:11:18 GMT

It is also worth to mention if you are using Syslog-ng PE or Rsyslog as relay, you can use the builtin disk plus memory buffer function to increase reliability without saving raw logs in files.

Re: Pros/cons of using Syslog-NG (or other syslog file receiver) vs. direct tcp/udp 514 to Splunk?

vladx — Tue, 31 Jul 2018 11:13:01 GMT

Link broken by the acquisition 😞