Metadata TRANSFORMS: Why is it not being applied after series of transforms?

[responsys_cm]

I have a customer with a nightmare syslog server environment -- different sourcetypes in different log files on different syslog servers, shared unqualified hostnames used in different data centers, some logs have FQDNs, some don't, etc.

My understanding is that the order of precedence for TRANSFORMS is source:: overwrites both sourcetype and host:: stanzas. host:: overwrites sourcetype stanzas.

So... I have TRANSFORMS stanzas applied to each source:: stanza to put the appropriate data into the correct sourcetype. I then apply index and host metadata TRANSFORMS to each of the sourcetype stanzas.

But for some reason, the host and index TRANSFORMS don't seem to get applied once an event has had a TRANSFORM applied in a source:: stanza. Is that expected behavior or are there limitations to metadata rewrites that they must occur only on the stanza with the highest precedence for a particular event?

[harsmarvania57]

Hi @responsys_cm,

The way you are trying to achieve I tried in my lab environment and it is not working which means props and transforms once applied then data will not route again through pipeline to modify host and index. However you can achieve sourcetype override, hostname override and index override with below configuration.

I have applied below configuration on WinEventLog:Security source and playing with EventCode 4688 to filter data using REGEX

props.conf

[source::WinEventLog:Security]
TRANSFORMS-sthostidx = test_st, test_host, test_idx

transforms.conf

[test_st]
DEST_KEY = MetaData:Sourcetype
REGEX = EventCode\=4688
FORMAT = sourcetype::test_sourcetype

[test_host]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = MetaData:Host
REGEX = test_sourcetype
FORMAT = host::testhost

[test_idx]
SOURCE_KEY = MetaData:Host
DEST_KEY = _MetaData:Index
REGEX = testhost
FORMAT = windows    

I hope this will help you.

Thanks,
Harshil

[youngsuh]

transform.conf 

[vmware:uag:admin]
REGEX = :\d\d\s+\w{5}\w{4}\suag-admin\:(.+)\n
FORMAT = sourcetype::vmware:uag:admin
DEST_KEY = MetaData:Sourcetype

[vmware:uag:audit]
REGEX = :\d\d\s+\w{5}\w{4}\suag-audit\:(.+)\n
FORMAT = sourcetype::vmware:uag:admin
DEST_KEY = MetaData:Sourcetype

[vmware:uag:esmanager]
REGEX = :\d\d\s+\w{5}\w{4}\suag-esmanager\:(.+)\n
FORMAT = sourcetype::vmware:uag:esmanager
DEST_KEY = MetaData:Sourcetype

Props.conf

[source::/var/log/$hostname$/syslog]
TRANSFORMS-sourcetype = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

[source::/var/log/$hostname$/syslog]
TRANSFORMS-sourcetype = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

[source::/var/log/$hostname$/syslog]
TRANSFORMS-sourcetype = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

@harsmarvania57  can you see anything wrong on any of .conf file?  it's not working.  I see no error too on splunkd.log

[cybermonday]

@harsmarvania57 - What is the need here to have [test_host] ?

can we directly define [test_idx] as below ?

[test_idx]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = _MetaData:Index
REGEX = test_sourcetype
FORMAT = windows 

[harsmarvania57]

I thought that OP wants to rename host as well sotest_host was provided. If you do not want to rename host then configuration provided by you should work for changing of index.