Splunk adds one header, then one more when forwarding to external logger.
Jan 29 14:09:01 host.localdomain: 2016 Jan 29 14:08:57 EST: %DAEMON-3-SYSTEM_MSG: error: setsockopt IP_TOS 16: Invalid argument: - sshd
Jan 29 14:09:01 host.localdomain Jan 29 14:09:01 host.localdomain : 2016 Jan 29 14:08:57 EST: %DAEMON-3-SYSTEM_MSG: error: setsockopt IP_TOS 16: Invalid argument: - sshd
Jan 29 17:00:01 host.localdomain Jan 29 17:00:01 host.localdomain : 2016 Jan 29 16:59:56 EST: %DAEMON-3-SYSTEM_MSG: error: setsockopt IP_TOS 16: Invalid argument: - sshd
Can you give us some information, ie, what is the configuration you are using to forward your data, and what kind of system is receiving it?
Hi Jeremiah, this log is from a Cisco switch sending syslog to SPLUNK.
We are also seeing multiple headers in logs from other systems as well coming in on source udp:514.
It appears as though SPLUNK is attaching another header before it goes out .
I have also attached all of the outputs stanzas at the end.
Below log is what's sent to external logger from another host. there are multiple headers again.
Jan 29 16:52:32 esx.mydomain Jan 29 16:52:32 esx.mydomain 2016-01-29T21:52:32.963Z ESX.MYDOMAINVpxa: [FF96FB90 verbose 'hostdstats']
Jan 29 16:52:32 esx.mydomain 2016-01-29T21:52:32.978Z ESX.MYDOMAIN Vpxa: [FF96FB90 verbose 'hostdstats'] Set internal stats for VM
[tcpout] maxQueueSize = 500KB forwardedindex.0.whitelist = .* forwardedindex.1.blacklist = _.* forwardedindex.2.whitelist = _audit forwardedindex.filter.disable = false indexAndForward = false autoLBFrequency = 30 blockOnCloning = true compressed = false disabled = false dropClonedEventsOnQueueFull = 5 dropEventsOnQueueFull = -1 heartbeatFrequency = 30 maxFailuresPerInterval = 2 secsInFailureInterval = 1 maxConnectionsPerIndexer = 2 forceTimebasedAutoLB = false sendCookedData = true connectionTimeout = 20 readTimeout = 300 writeTimeout = 300 useACK = false #defaultGroup=nowhere [syslog] defaultGroup = [syslog:Everything] disabled = true timestampformat = %b %e %H:%M:%S server = x.x.x.x:514 [syslog:ext_logger] disabled = false timestampformat = %b %e %H:%M:%S server = x.x.x.x:514
So, you may need to set:
syslogSourceType = <string>
In your outputs.conf syslog stanza. The string value should match the sourcetype of your Cisco data, so that Splunk knows this is syslog data and doesn't need to add a timestamp/hostname to the beginning of the log entry.
"Data which does not match the rules has a header, optionally a timestamp (if defined in 'timestampformat'), and a hostname added to the front of the event. This is how Splunk causes arbitrary log data to match syslog expectations."
There is a Splunk wiki article that might help explain what is happening when your data is being processed and passed on to a syslog destination:
It sounds like you are passing data directly to splunk via syslog. I prefer to have a syslog server (syslog-ng or rsyslog) setup to receive my syslog data and write to a file. Then I use a Splunk forwarder to read the files and forward them to my indexer. This also gives you the advantage of routing data directly via syslog-ng if you need to. There's a discussion of the pros/cons here: