Can I set-up Splunk to replace a syslog server?

[twinspop]

We need to ingest syslog data. Rather then send to a syslog server, then read data from disk with a Forwarder, it seems like sending directly to a Forwarder listening on port 514 would be more efficient. Are there any problems with doing this?

[twinspop · Accepted solution]

No. Don't do it. Here's my story.

Our COVID19 work from home barage started up and our execs wanted VPN stats pronto. Security guys said they'd point their syslog at where we wanted. I quickly built a VM and threw a UF on it. Next, I set-up a UDP input on 514, and configured the props, indexes, etc. Finally I lit it up, boom! Data coming in.

For a few days we worked on reports. Then I started noticing missing events here and there. As I dug in I found duckfez's post on tracking UDP errors. And oh man were there a lot of errors. Like thousands per second. We're definitely dropping events.

I set-up rsyslog to receive instead. It's writing to disk, and splunk is reading from there. I also set-up logrotate to clean up cuz these logs are gonna be big. With some tweaking of my props and transforms, I got everything to match the slightly different appearance of the logs.

End results with the exact same stream of data being thrown at the server:

While using Splunk to receive directly:

2,500 events/sec
10,000 UDP rcv buf errors/sec

While using rsyslog to receive, and Splunk reads from disk:

25,000 events/sec
0 UDP rcv buf errors/sec

No other changes were made to the host or the log stream being shoved at it.

Don't use Splunk to receive syslog.