Precedence issues between host and sourcetype

[scottjreynolds]

In our environment we have two subdomains which can be assigned to any particular server:

bo-prod.domain.com & prod.domain.com

Due to limitations with syslog we have written transform rules to append bo-prod.domain.com to the shortened hostname or our servers as bo-prod.domain.com is the domain for 95% of our servers. We're now rolling out the prod.domain.com servers and as you've probably guess, syslog entries are being improperly updated for host::webserver1* with bo-prod.domain.com instead of prod.domain.com regardless of the rules in place below.

Here are the current configuration entries we have in place on the indexer to remedy the issue

props.conf

[host::webserver1]
TRANSFORMS-syslog_add_fqdn_prod = syslog_add_fqdn_prod
priority = 101
[host::webserver1.bo-prod.domain.com]
TRANSFORMS-syslog_add_fqdn_prod = syslog_add_fqdn_prod
priority = 101
[host::webserver1.prod.domain.com]
TRANSFORMS-syslog_add_fqdn_prod = syslog_add_fqdn_prod
priority = 101

[syslog]
TRANSFORMS-syslog_add_fqdn_bo-prod = syslog_add_fqdn_bo-prod
[linux_audit]
TRANSFORMS-syslog_add_fqdn_bo-prod = syslog_add_fqdn_bo-prod
[linux_secure]
TRANSFORMS-syslog_add_fqdn_bo-prod = syslog_add_fqdn_bo-prod
[mysqld]
TRANSFORMS-syslog_add_fqdn_bo-prod = syslog_add_fqdn_bo-prod
[postfix_syslog]
TRANSFORMS-syslog_add_fqdn_bo-prod = syslog_add_fqdn_bo-prod
[sendmail_syslog]
TRANSFORMS-syslog_add_fqdn_bo-prod = syslog_add_fqdn_bo-prod

transforms.conf

[syslog_add_fqdn_bo-prod]
DEST_KEY = MetaData:Host
SOURCE_KEY = MetaData:Host
REGEX = host::([A-Za-z][-_A-Za-z0-9]*[A-Za-z0-9])$
FORMAT = host::$1.bo-prod.domain.com

[syslog_add_fqdn_prod]
DEST_KEY = MetaData:Host
SOURCE_KEY = MetaData:Host
REGEX = host::([A-Za-z][-_A-Za-z0-9]*[A-Za-z0-9])$
FORMAT = host::$1.prod.domain.com

Based on the Splunk documentation it sounds like a host spec will always be processed first if a chunk of data matches both a host and sourcetype spec so i'm not sure why this behaviour is occurring.


For settings that are specified in multiple categories of matching [] stanzas, [host::] settings override [] settings. Additionally, [source::] settings override both [host::] and [] settings.


[Drainy]

From my experience you can share the same name between props stanzas without issue. However what I would ask is what is the name of the Splunk server? Your override is designed to change the host metadata field that is indexed. Your transform however is trying to override it after it appears to be defined.

I would try to apply the host transform to the source (so instead of host:: do a source:) so that the transform is applied as early on as possible.

[maverick]

I may be wrong here, but I think the name extension of your custom transforms all need to be different to be considered valid. Also, according to the docs, I do not think a "priority" is affective between host and sourcetype stanzas. And since you have a literal match and not a pattern match in your host stanzas, I do not think anything equal to or above a value of 100 will work anyway.

Therefore, you might try the following and see if it resolves the issue you are seeing.

[host::webserver1]
  TRANSFORMS-syslog_add_fqdn_prod1 = syslog_add_fqdn_prod
[host::webserver1.bo-prod.domain.com]
  TRANSFORMS-syslog_add_fqdn_prod2 = syslog_add_fqdn_prod
[host::webserver1.prod.domain.com]
  TRANSFORMS-syslog_add_fqdn_prod3 = syslog_add_fqdn_prod

[syslog]
  TRANSFORMS-syslog_add_fqdn_bo-prod1 = syslog_add_fqdn_bo-prod
[linux_audit]
  TRANSFORMS-syslog_add_fqdn_bo-prod2 = syslog_add_fqdn_bo-prod
[linux_secure]
  TRANSFORMS-syslog_add_fqdn_bo-prod3 = syslog_add_fqdn_bo-prod
[mysqld]
  TRANSFORMS-syslog_add_fqdn_bo-prod4 = syslog_add_fqdn_bo-prod
[postfix_syslog]
  TRANSFORMS-syslog_add_fqdn_bo-prod5 = syslog_add_fqdn_bo-prod
[sendmail_syslog]
  TRANSFORMS-syslog_add_fqdn_bo-prod6 = syslog_add_fqdn_bo-prod

Anyway, hope this helps. Reply back if it does or not and someone here will help you figure it out.