All,
What's the current process for onboarding RFC5425 (SYslog with TLS) logging? I see docs from 2013 or so recommending inputs.conf with a cert and some docs saying just rely on syslog-ng's facility. How ever nothing modern, I would assume this functionality moved to HEC but I can't find a doc saying so?
If someone could point me to a modern doc and a walk through it would be appreciated.
Switching to using rsyslog or syslog-ng and HEC is the method that I have used for several instances. It doesn't require the disk resources that a universal forwarder does, and so it can have higher throughput (at least double, as I have had 1.2TB/day running through a server that would be choking at half that amount with a UF. George's article is great, though there are some more recent articles about it and also a .conf2017 presentation on the topic:
http://conf.splunk.com/sessions/2017-sessions.html#search=aggregated%20&
Is syslog-ng supported by Splunk? That is if there is an issue at 3am and my NOC calls, will they help restore that service? My knee jerk is to use Splunk support methods only.
No, it is not supported. They may help you. You can use splunkd, but there are a number of problems with doing that. Top on my lis is that every time you restart Splunk you will loose any data going to it until it is back up. With using syslog, restarts of that service take usually less than a second, so you rarely loose much. If you feel that your data on restarts is not that important, or you never restart splunk for upgrades, config changes, patching, maintenance of any type, etc., and you want Splunk support to be there for you, then use Splunk. BTW, Splunk Support is great, but you could easily wait 2 hours for their support even if it is a P1, and longer if it is a P2 case. You have to weigh the benefits and shortcomings and do what is best for your situation.
There are more reasons, and they are mentioned in George Starcher's article. Make sure you read it.
If we are just talking syslog, then I do it like this (although many people are switching to HEC, queue @cpetterborg):
http://www.georgestarcher.com/splunk-success-with-syslog/
Sorry I wasn't very clear. That is very similar to what we're doing with syslog-ng, but I am mostly looking for what Splunk recommends for Syslog over TLS and a walk through on how to do it as I have never set it up before. I want to be sure when we call support for help at 3am we don't have any trouble.
The following might be useful - Log monitoring and analysis with rsyslog and Splunk