Getting Data In

Can I set-up Splunk to replace a syslog server?

twinspop
Influencer

We need to ingest syslog data. Rather then send to a syslog server, then read data from disk with a Forwarder, it seems like sending directly to a Forwarder listening on port 514 would be more efficient. Are there any problems with doing this?

0 Karma
1 Solution

twinspop
Influencer

No. Don't do it. Here's my story.

Our COVID19 work from home barage started up and our execs wanted VPN stats pronto. Security guys said they'd point their syslog at where we wanted. I quickly built a VM and threw a UF on it. Next, I set-up a UDP input on 514, and configured the props, indexes, etc. Finally I lit it up, boom! Data coming in.

For a few days we worked on reports. Then I started noticing missing events here and there. As I dug in I found duckfez's post on tracking UDP errors. And oh man were there a lot of errors. Like thousands per second. We're definitely dropping events.

I set-up rsyslog to receive instead. It's writing to disk, and splunk is reading from there. I also set-up logrotate to clean up cuz these logs are gonna be big. With some tweaking of my props and transforms, I got everything to match the slightly different appearance of the logs.

End results with the exact same stream of data being thrown at the server:

While using Splunk to receive directly:

2,500 events/sec
10,000 UDP rcv buf errors/sec

While using rsyslog to receive, and Splunk reads from disk:

25,000 events/sec
0 UDP rcv buf errors/sec

No other changes were made to the host or the log stream being shoved at it.

Don't use Splunk to receive syslog.

View solution in original post

twinspop
Influencer

No. Don't do it. Here's my story.

Our COVID19 work from home barage started up and our execs wanted VPN stats pronto. Security guys said they'd point their syslog at where we wanted. I quickly built a VM and threw a UF on it. Next, I set-up a UDP input on 514, and configured the props, indexes, etc. Finally I lit it up, boom! Data coming in.

For a few days we worked on reports. Then I started noticing missing events here and there. As I dug in I found duckfez's post on tracking UDP errors. And oh man were there a lot of errors. Like thousands per second. We're definitely dropping events.

I set-up rsyslog to receive instead. It's writing to disk, and splunk is reading from there. I also set-up logrotate to clean up cuz these logs are gonna be big. With some tweaking of my props and transforms, I got everything to match the slightly different appearance of the logs.

End results with the exact same stream of data being thrown at the server:

While using Splunk to receive directly:

2,500 events/sec
10,000 UDP rcv buf errors/sec

While using rsyslog to receive, and Splunk reads from disk:

25,000 events/sec
0 UDP rcv buf errors/sec

No other changes were made to the host or the log stream being shoved at it.

Don't use Splunk to receive syslog.

Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Event Series May & June: From Network Visibility to Service Intelligence

Unifying the Network: Moving from Alert Noise to Service Intelligence with Splunk ITSI In today’s hybrid ...

Global Splunk User Group Events: May + June 2026

Your Splunk Community Awaits: Discover Upcoming User Group Events Worldwide    Staying ahead in the fast-paced ...

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas

Why Splunk Customers Should Attend Cisco Live 2026 Las Vegas     Cisco Live 2026 is almost here, and this ...