SYSLOG_ROUTING ISSUE

[dshakespeare_sp]

Customer has the audit.log output to a separate logging system for security team use.
The security teams app expects a standard format. Customer has noted that on some systems they are sending their hostname prepended to the event and on others they aren’t. For example

Indexer 1
05-24-2013 10:56:02.730 +0000 INFO AuditLogger - Audit:[timestamp=05-24-2013 10:56:02.730, user=n/a, action=splunkShuttingDown, info=n/a][n/a]

SH
09-02-2013 09:05:01.756 +0000 INFO AuditLogger - Audit:[timestamp=09-02-2013 09:05:01.756, user=splunk-system-user, action=quota, info=user=splunk-system-user, seach_id=scheduler_nobodyapp_RMD5f280a649bde8003c_at_1378112700_353782, elapsed_ms=11, cache_size=218][n/a]

Indexer 2
qa2si1 09-02-2013 09:06:03.208 +0000 INFO AuditLogger - Audit:[timestamp=09-02-2013 09:06:03.208, user=splunk-system-user, action=search, info=granted REST: /search/jobs/remote_qa2sh1_scheduler_nobodyapp_RMD5f280a649bde8003c_at_1378112760_353795/search.log][n/a]

Indexer 3
qa2si2 09-02-2013 09:06:03.314 +0000 INFO AuditLogger - Audit:[timestamp=09-02-2013 09:06:03.314, user=splunk-system-user, action=search, info=granted REST: /search/jobs/remote_qa2sh1_scheduler_nobodyapp_RMD582a3d2154207a5d0_at_1378112760_353796/search.log][n/a]

As you can see, the hostname is prepended to the start of two events, not the others. These are four different Splunk instances.

Customer has run TCPDUMP on all hosts and verified that on some they are leaving with the hostname but not on others.

Customer originally thought that this was a DNS resolution issue - this was not the case

I have checked diags between a working / non working machine - Everything looks identical

Inputs.conf
[monitor://$SPLUNK_HOME/var/log/splunk]
blacklist = audit.log(.\d+)?
index = _internal

[monitor://$SPLUNK_HOME/var/log/splunk/audit.log]
index = _audit
sourcetype = audit_logs

Props.conf
[audit_logs]
SEDCMD-removenewline = s/\n//g
TRANSFORMS-syslog = syslog_routing

Transforms.conf
[syslog_routing]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_out

Outputs.conf
[syslog:syslog_out]
disabled = false
server = xx.xx.xx.xx:514
type = udp

DNS resolution is sams on all systems
They all have the same /etc/hosts file too

All builds come from a prepackaged SuSe Linux System

Why are they different and where does Splunk pull the host from?

Splunk version is 5.0.2
I note there were some _SYSLOG_ROUTING issues in 5.0.2 resolved in 5.0.4 - bit they done seem to apply to this senario and I would expect identical behaviour (either all with hostname prepended or not)

Shaky

[Masa]

In general, when event's sourcetype is not syslog, Splunk add host. Maybe some splunk configuration is overwriting the customer configuration(?) Or, because audit.log is splunk internal log,

My suggestion is to use "syslogSourcetype" attribute in outputs.conf.

The following link has additional information for general Splunk Syslog Routing
http://wiki.splunk.com/Community:Test:How_Splunk_behaves_when_receiving_or_forwarding_udp_data

If it is a Support case, it is a good idea to double-check configurations with btool output, and check splunkd.log.