Using Splunk to collect Syslog and forward to remote syslog

[michael_lee]

So Splunk can collect syslog by configure data input at TCP/UDP port 514. Can I know:

• Splunk does not manipulate the syslog data coming in right?
• How then to forward these syslog data to another remote syslog server? Splunk indexes them as they come in through port 514 so I don't think spunk can forward to a remote syslog server within spunk itself.?
• I am totally using spunk as syslog collector in this situation. No rsyslog or syslog-ng

thanks

Update: I realize some of my logs could not be converted to syslog format, hence I am still going to try going ahead with Splunk as the syslog collector. I am just using a dedicated splunk instance as the syslog indexer and will not have too much restarting done. Even if there is a restart, my Splunk forwarders can store/buffer events first before sending..I think.

[rfaircloth_splu]

Splunk CAN forward syslog however this should be avoided in almost all cases. Splunk processes reload or restart for a number of reasons and are not designed to be HA for syslog. There are cases such as small/remote office where this is an appropriate use for Splunk, not the rule however.

Syslog-NG is the most common and preferred aggregation solution in front of Splunk. Generally speaking a NLB (or clustered pair) will be placed in front of two or more syslog servers. Syslog-NG will write a copy of the data to disk for the Universal Forwarder to collect and forward a subset of messages to another system such as the Cisco NAM or UniCenter for It Monitoring.

My guide for syslog configuration would be a good starting point for you
http://www.rfaircloth.com/2016/01/17/building-reliable-syslog-infrastructure-on-centos-7/

[jkat54 · Accepted solution]

UPDATE:

YOU CAN FORWARD SYSLOG IN ORIGINAL FORMAT FROM A HEAVY FORWARDER:
http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Forwarddatatothird-partysystemsd#Syslog...

Old WRONG answer:
Splunk manipulates the syslog data for sure. It changes it into the indexed form of data and compresses the data for storage. If you go look at the index file, it will be binary... not syslog events.

You cannot forward syslog from splunk. You can however pull data out of splunk using ODBC drivers, python, bash scripting, etc.

If you need to forward syslog, you'll need to stick to traditional methods such as syslog-ng, rsyslog, kafka, redis, network load balancing, etc.

[rfaircloth_splu]

I downvoted this post because the solution proposed would be unstable for production use.

[jkat54]

If Splunk can't support this as a "production stable" functionality then it shouldn't be in the product IMHO. Regardless the questions asked were answered. Down vote all you want. We both know Splunk isn't designed to be a syslog forwarder.

[a212830]

Pretty sure Splunk can forward syslog (as syslog) to other sources - it's just done at the forwarding layer (and might require a HFW). Not sure that I'd recommend it as a best practice, but it is possible.

[jkat54]

A Splunk forwarder forwards "cooked" events by default. Cooked events will not be in syslog format.

I never realized it but you CAN forward traditional syslog. SORRY! EDITED MY ANSWER.

http://docs.splunk.com/Documentation/Splunk/6.2.0/Forwarding/Forwarddatatothird-partysystemsd#Syslog...

[jkat54]

Also if you're listening on port 514 with splunk on a linux machine, then that means you're most likely running splunk as root. That is against best practices. Consider yourself warned.

[michael_lee]

ok then the only solution is to use rsyslog and then use a forwarder and configure output.conf to forward to a remote. thanks.