Can I set-up Splunk to replace a syslog server?

twinspop — Fri, 27 Mar 2020 20:05:23 GMT

We need to ingest syslog data. Rather then send to a syslog server, then read data from disk with a Forwarder, it seems like sending directly to a Forwarder listening on port 514 would be more efficient. Are there any problems with doing this?

Re: Can I set-up Splunk to replace a syslog server?

twinspop — Fri, 27 Mar 2020 20:06:09 GMT

No. Don't do it. Here's my story.

Our COVID19 work from home barage started up and our execs wanted VPN stats pronto. Security guys said they'd point their syslog at where we wanted. I quickly built a VM and threw a UF on it. Next, I set-up a UDP input on 514, and configured the props, indexes, etc. Finally I lit it up, boom! Data coming in.

For a few days we worked on reports. Then I started noticing missing events here and there. As I dug in I found duckfez's post on tracking UDP errors. And oh man were there a lot of errors. Like thousands per second. We're definitely dropping events.

I set-up rsyslog to receive instead. It's writing to disk, and splunk is reading from there. I also set-up logrotate to clean up cuz these logs are gonna be big. With some tweaking of my props and transforms, I got everything to match the slightly different appearance of the logs.

End results with the exact same stream of data being thrown at the server:

While using Splunk to receive directly:

2,500 events/sec
10,000 UDP rcv buf errors/sec

While using rsyslog to receive, and Splunk reads from disk:

25,000 events/sec
0 UDP rcv buf errors/sec

No other changes were made to the host or the log stream being shoved at it.

Don't use Splunk to receive syslog.

topic Re: Can I set-up Splunk to replace a syslog server? in Getting Data In

Can I set-up Splunk to replace a syslog server?

Re: Can I set-up Splunk to replace a syslog server?