What's the difference between ingesting TCP as TCP vs. Syslog TCP

[spl_aficionado]

We recently found out that we couldn't send TCP data as Syslog because it didn't have the proper header, but streaming it as TCP works perfectly fine. What are the best practices surrounding a native TCP port vs. Syslog? 

[PickleRick]

Well... it's a topic which seems easy but gets more and more complicated the further you dig into it.

Firstly, there is no single thing called "syslog".

Syslog used to be a "standardized" way of logging events in unices but since than grew significantly and can mean anything:

- Anything sent to port 514/udp or tcp

- Anything sent directly over a network stream without using any fancy protocol overhead (usually just separating items with new line characters)

- Anything sent over UDP - one item per datagram

- Anything following RFC 3164 (BSD syslog) format or RFC 5424

So right from the start without being much more precise than just saying "syslog" it's difficult to say what we're talking about.

What do you mean by "sending TCP data as syslog" vs. "streaming as TCP"? What source solution are we talking about? How are you trying to receive and process the events?

[isoutamo]

I think that the big question is “can I lost events?” Then how much I can lost? If the amount is reasonable then follow what @diogofgm said. If you are thinking that you can’t lose any events and you still are using syslog, please think again. Then follow @PickleRick ‘s thinking. In reality using syslog as method to deliver events without any losses it’s really hard almost impossible. I know that @PickleRick will said that it’s impossible 😉 IMHO it’s depends on your architecture and setups, but at least in some cases it’s doable maybe not all cases?

[PickleRick]

Generally, having a fully 100% proof event delivery is extremelly difficult unless you are ready to have high probability of duplicated events. With syslog reliable delivery generally doesn't exist.

With UDP the source doesn't have any way to determine if the event has been received or not. So UDP is not reliable by design.

With TCP it _could_ be more reliable (and there are some solutions which do keep track of whether the destination actually received the event at least on the TCP stream level - most notable one being Checkpoint's Log Exporter which can get stuck for quite a long time if there is noone to receive the events) but typically the sources don't care about the destination. There is just a simple send() routine within the source's event handling process and as soon as the event is passed there the main process doesn't care anymore. If the connection to the destination can be established the event will be sent. If it cannot it won't. And that's fine because syslog was always meant to be simple, robust and easy to implement and maintain.

So yes, there is no way to make a 100% reliable syslog delivery. (One can argue that there is no way to make 100% reliable delivery because every component which would have to queue events in case of downstream path failure could get overfilled but that's another thing - just physical hardware limitations).

[diogofgm]

The best practice is actually not sending syslog directly to splunk as you'll most likely lose data. You should instead use something like a syslog server (rsyslog, syslog-ng, etc.) to receive the data and then you have many options to send the data to splunk (e.g. Splunk UF). 

Splunk has SC4S (Splunk Connect for Syslog) that can handle receiving and forwarding data to Splunk which is supported by Splunk.