I'm looking for guidance about a logging problem I am trying to solve. Right now we have a few security onion boxes sending snort logs to both our log server and to Splunk using syslog-ng. This works fine.
The powers that be now want to remove the direct send to Splunk and just pipe the logs from the syslog box into Splunk. What I would like to do is just forward these specific log files which are under /var/log/remote/IP1, /var/log/remote/IP2 to the Splunk box. Is there an easy way to accomplish this or do I need to get cute with eventtypes..etc? Hopefully that makes sense.
Piping syslog through an intermediate server is accepted Best Practice to avoid losing data.
If you have the UF on your syslog collector, just configure your inputs to monitor those files. Example inputs.conf:
[monitor:///var/log/remote/IP1] host=IP1 sourcetype=syslog index=syslogidx [monitor:///var/log/remote/IP2] host=IP2 sourcetype=syslog index=syslogidx
If you need something more sophisticated, explain that scenario.
I ended up setting up a forward in rsyslog. So it logs into /var/log/remote/ip.log then forwards to Splunk.
if $fromhost-ip=='192.168.211.2' then @192.168.211.3:514
Is it better to use the universal forwarder?
Yes, the universal forwarder is better. It handles retry, throttling, and more that rsyslog doesn't.
Most definitely I agree. It's too easy to loose syslog data. The forwarder on the syslog server is the best way to go.