Why are we seeing duplicate headers? (host and timestamp)

[jppham]

Splunk adds one header, then one more when forwarding to external logger.

SPLUNK entry
Jan 29 14:09:01 host.localdomain: 2016 Jan 29 14:08:57 EST: %DAEMON-3-SYSTEM_MSG: error: setsockopt IP_TOS 16: Invalid argument: - sshd[6771]

External logger
Jan 29 14:09:01 host.localdomain Jan 29 14:09:01 host.localdomain : 2016 Jan 29 14:08:57 EST: %DAEMON-3-SYSTEM_MSG: error: setsockopt IP_TOS 16: Invalid argument: - sshd[6771]

RAW –tcpdump
Jan 29 17:00:01 host.localdomain Jan 29 17:00:01 host.localdomain : 2016 Jan 29 16:59:56 EST: %DAEMON-3-SYSTEM_MSG: error: setsockopt IP_TOS 16: Invalid argument: - sshd[12046]

[Jeremiah]

So, you may need to set:

syslogSourceType = <string>

In your outputs.conf syslog stanza. The string value should match the sourcetype of your Cisco data, so that Splunk knows this is syslog data and doesn't need to add a timestamp/hostname to the beginning of the log entry.

From http://docs.splunk.com/Documentation/Splunk/6.3.2/admin/Outputsconf:

"Data which does not match the rules has a header, optionally a timestamp (if defined in 'timestampformat'), and a hostname added to the front of the event. This is how Splunk causes arbitrary log data to match syslog expectations."

There is a Splunk wiki article that might help explain what is happening when your data is being processed and passed on to a syslog destination:

https://wiki.splunk.com/Community:Test:How_Splunk_behaves_when_receiving_or_forwarding_udp_data

It sounds like you are passing data directly to splunk via syslog. I prefer to have a syslog server (syslog-ng or rsyslog) setup to receive my syslog data and write to a file. Then I use a Splunk forwarder to read the files and forward them to my indexer. This also gives you the advantage of routing data directly via syslog-ng if you need to. There's a discussion of the pros/cons here:

https://answers.splunk.com/answers/103295/pros-cons-of-using-syslog-ng-or-other-syslog-file-receiver...

[Jeremiah]

Can you give us some information, ie, what is the configuration you are using to forward your data, and what kind of system is receiving it?

[jppham]

Hi Jeremiah, this log is from a Cisco switch sending syslog to SPLUNK.
We are also seeing multiple headers in logs from other systems as well coming in on source udp:514.
It appears as though SPLUNK is attaching another header before it goes out .

I have also attached all of the outputs stanzas at the end.

Below log is what's sent to external logger from another host. there are multiple headers again.

external logger
Jan 29 16:52:32 esx.mydomain Jan 29 16:52:32 esx.mydomain 2016-01-29T21:52:32.963Z ESX.MYDOMAINVpxa: [FF96FB90 verbose 'hostdstats']

SPLUNK
Jan 29 16:52:32 esx.mydomain 2016-01-29T21:52:32.978Z ESX.MYDOMAIN Vpxa: [FF96FB90 verbose 'hostdstats'] Set internal stats for VM

OUTPUTS:

[tcpout]
maxQueueSize = 500KB
forwardedindex.0.whitelist = .*
forwardedindex.1.blacklist = _.*
forwardedindex.2.whitelist = _audit
forwardedindex.filter.disable = false
indexAndForward = false
autoLBFrequency = 30
blockOnCloning = true
compressed = false
disabled = false
dropClonedEventsOnQueueFull = 5
dropEventsOnQueueFull = -1
heartbeatFrequency = 30
maxFailuresPerInterval = 2
secsInFailureInterval = 1
maxConnectionsPerIndexer = 2
forceTimebasedAutoLB = false
sendCookedData = true
connectionTimeout = 20 
readTimeout = 300
writeTimeout = 300 
useACK = false
#defaultGroup=nowhere

[syslog]
defaultGroup = 

[syslog:Everything]
disabled = true
timestampformat = %b %e %H:%M:%S
server = x.x.x.x:514

[syslog:ext_logger]
disabled = false
timestampformat = %b %e %H:%M:%S
server = x.x.x.x:514