As I'm currently engaged on an external SOC onboarding project, I've been quite involved in adopting the forwarding layer that has been made for cloning input feeds and streaming output via syslog to the SOC as well as to our indexers.
One key issue has arose - the HF/IUF always sets the host field in the message being relayed, to itself... Then appears to embed the message.
This behavior has caused us to go away and create a separate syslog-ng cluster for sending to the SoC, then use file dumps to get a LF to pick this up and send to splunk.
This is tedious, and what I'm asking/suggesting is what customizations are available for the BSD syslog protocol usage in splunk forwarding?
I've almost lost my wits with this and for now I have wrote-off the splunk forwarding layer someone else designed, I'm aware of this being reported on various posts but nobody has explored this properly, nobody has suggested a development request with splunk to make syslog outputs more tune-able.
I like the idea of being able to specify the sourcetype details in the header etc using HEC.. It does look a bit more messier though than just using syslog-ng and inputs.conf...
I appreciate HTTP is flexible but what other benefits do I get by using HEC that I can't get using inputs.conf with a LF?
splunk really need to consider more customization of their syslog stuff, it just seems like input/output syslog only, no other options, surely this could be more tunable it would save so much hassle.
If you don't want to get into the HEC solution also posted here, the best practice is not to receive on a Splunk port for syslog. Setup syslog-ng or rsyslog, write to file and use UF to pick them up. Then you can reliably send the desired systems to the right index and sourcetype. http://www.georgestarcher.com/splunk-success-with-syslog/
Too much parsing of the data and rewriting in the processing queues can back things up.
Hi, yes that's what I'm proposing but there should be more options. HEC would be a good standard if everything talked HEC, it doesn't though.
Syslog is becoming far more important to be integrated with various things, SIEM, splunk, ELK, internal and external SOC etc.... Perhaps HEC should replace it in future?
I don't think splunk are taking syslog seriously enough to say that their whole platform is perfect to assist with routing of events all over the place, but over-writes the host field with its own instance name, just to ruin the show!
It's a massive let-down considering splunk professional services were once here and built us a forwarding layer for syslog - that wont do the job.
I've seen others talk of it but nobody has suggested doing anything about it, instead we have to use the long way around, and even worse we have concluded that syslog agents will completely replace splunk LF's, requiring a different solution to get visibility of the assets.