The recommend best practice for your case and pretty much all syslog setups is to send the syslogs to a central server, where a syslog daemon (preferrably syslog-ng, or rsyslog) collects all data, and writes it to disk, split by hostname, date, etc.
You can then have a UF (or HF, if necessary) monitor those files and forward them to an indexer.
This is best practice for a bunch of reasons:
- Less prone to data loss, because syslog daemons are less frequently restarted, and restart much faster than Splunk
- Less prone to data loss, because all data is immediatly written to disk, where it can be picked up by Splunk
- better performance, because syslog daemons are better at ingesting syslog data than Splunk TCP/UDP inputs
In this case, a network failure between UF/HF and IDX won't cause any problems, because the data is still being written to disk - UF/HF will just continue to forward it when the connection is available again. I'd prefer this over Splunk persistent queues, because it also allows for much easier troubleshooting - because the data is written to disk, and you can look at it. If something is wrong, you can check if you got bad data, or if something went wrong after that with your Splunk parsing.
You can find more details in the .conf 2017 recording/slides for "The Critical Syslog Tricks That No One Seems to Know About", or in the blog posts Using Syslog-ng with Splunk and Splunk Success with Syslog.
Hope that helps - if it does I'd be happy if you would upvote/accept this answer, so others could profit from it. 🙂
... View more
