Customer has the audit.log output to a separate logging system for security team use.
The security teams app expects a standard format. Customer has noted that on some systems they are sending their hostname prepended to the event and on others they aren’t. For example
Indexer 1
05-24-2013 10:56:02.730 +0000 INFO AuditLogger - Audit:[timestamp=05-24-2013 10:56:02.730, user=n/a, action=splunkShuttingDown, info=n/a][n/a]
SH
09-02-2013 09:05:01.756 +0000 INFO AuditLogger - Audit:[timestamp=09-02-2013 09:05:01.756, user=splunk-system-user, action=quota, info=user=splunk-system-user, seach_id=scheduler_nobodyapp_RMD5f280a649bde8003c_at_1378112700_353782, elapsed_ms=11, cache_size=218][n/a]
Indexer 2
qa2si1 09-02-2013 09:06:03.208 +0000 INFO AuditLogger - Audit:[timestamp=09-02-2013 09:06:03.208, user=splunk-system-user, action=search, info=granted REST: /search/jobs/remote_qa2sh1_scheduler_nobodyapp_RMD5f280a649bde8003c_at_1378112760_353795/search.log][n/a]
Indexer 3
qa2si2 09-02-2013 09:06:03.314 +0000 INFO AuditLogger - Audit:[timestamp=09-02-2013 09:06:03.314, user=splunk-system-user, action=search, info=granted REST: /search/jobs/remote_qa2sh1_scheduler_nobodyapp_RMD582a3d2154207a5d0_at_1378112760_353796/search.log][n/a]
As you can see, the hostname is prepended to the start of two events, not the others. These are four different Splunk instances.
Customer has run TCPDUMP on all hosts and verified that on some they are leaving with the hostname but not on others.
Customer originally thought that this was a DNS resolution issue - this was not the case
I have checked diags between a working / non working machine - Everything looks identical
Inputs.conf
[monitor://$SPLUNK_HOME/var/log/splunk]
blacklist = audit.log(.\d+)?
index = _internal
[monitor://$SPLUNK_HOME/var/log/splunk/audit.log]
index = _audit
sourcetype = audit_logs
Props.conf
[audit_logs]
SEDCMD-removenewline = s/\n//g
TRANSFORMS-syslog = syslog_routing
Transforms.conf
[syslog_routing]
REGEX = .
DEST_KEY = _SYSLOG_ROUTING
FORMAT = syslog_out
Outputs.conf
[syslog:syslog_out]
disabled = false
server = xx.xx.xx.xx:514
type = udp
DNS resolution is sams on all systems
They all have the same /etc/hosts file too
All builds come from a prepackaged SuSe Linux System
Why are they different and where does Splunk pull the host from?
Splunk version is 5.0.2
I note there were some _SYSLOG_ROUTING issues in 5.0.2 resolved in 5.0.4 - bit they done seem to apply to this senario and I would expect identical behaviour (either all with hostname prepended or not)
Shaky
