Splunk Search

Metadata TRANSFORMS: Why is it not being applied after series of transforms?

responsys_cm
Builder

I have a customer with a nightmare syslog server environment -- different sourcetypes in different log files on different syslog servers, shared unqualified hostnames used in different data centers, some logs have FQDNs, some don't, etc.

My understanding is that the order of precedence for TRANSFORMS is source:: overwrites both sourcetype and host:: stanzas. host:: overwrites sourcetype stanzas.

So... I have TRANSFORMS stanzas applied to each source:: stanza to put the appropriate data into the correct sourcetype. I then apply index and host metadata TRANSFORMS to each of the sourcetype stanzas.

But for some reason, the host and index TRANSFORMS don't seem to get applied once an event has had a TRANSFORM applied in a source:: stanza. Is that expected behavior or are there limitations to metadata rewrites that they must occur only on the stanza with the highest precedence for a particular event?

Labels (1)
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @responsys_cm,

The way you are trying to achieve I tried in my lab environment and it is not working which means props and transforms once applied then data will not route again through pipeline to modify host and index. However you can achieve sourcetype override, hostname override and index override with below configuration.

I have applied below configuration on WinEventLog:Security source and playing with EventCode 4688 to filter data using REGEX

props.conf

[source::WinEventLog:Security]
TRANSFORMS-sthostidx = test_st, test_host, test_idx

transforms.conf

[test_st]
DEST_KEY = MetaData:Sourcetype
REGEX = EventCode\=4688
FORMAT = sourcetype::test_sourcetype

[test_host]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = MetaData:Host
REGEX = test_sourcetype
FORMAT = host::testhost

[test_idx]
SOURCE_KEY = MetaData:Host
DEST_KEY = _MetaData:Index
REGEX = testhost
FORMAT = windows    

I hope this will help you.

Thanks,
Harshil

youngsuh
Contributor

transform.conf 

[vmware:uag:admin]
REGEX = :\d\d\s+\w{5}\w{4}\suag-admin\:(.+)\n
FORMAT = sourcetype::vmware:uag:admin
DEST_KEY = MetaData:Sourcetype

[vmware:uag:audit]
REGEX = :\d\d\s+\w{5}\w{4}\suag-audit\:(.+)\n
FORMAT = sourcetype::vmware:uag:admin
DEST_KEY = MetaData:Sourcetype

[vmware:uag:esmanager]
REGEX = :\d\d\s+\w{5}\w{4}\suag-esmanager\:(.+)\n
FORMAT = sourcetype::vmware:uag:esmanager
DEST_KEY = MetaData:Sourcetype

 

Props.conf

[source::/var/log/$hostname$/syslog]
TRANSFORMS-sourcetype = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

[source::/var/log/$hostname$/syslog]
TRANSFORMS-sourcetype = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

[source::/var/log/$hostname$/syslog]
TRANSFORMS-sourcetype = vmware:uag:admin, vmware:uag:audit, vmware:uag:esmanager

@harsmarvania57  can you see anything wrong on any of .conf file?  it's not working.  I see no error too on splunkd.log

 

0 Karma

cybermonday
Explorer

@harsmarvania57 - What is the need here to have [test_host] ?

can we directly define [test_idx] as below ?

[test_idx]
SOURCE_KEY = MetaData:Sourcetype
DEST_KEY = _MetaData:Index
REGEX = test_sourcetype
FORMAT = windows 
0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

I thought that OP wants to rename host as well sotest_host was provided. If you do not want to rename host then configuration provided by you should work for changing of index.

0 Karma
Get Updates on the Splunk Community!

Index This | A sphere has three, a circle has two, and a point has zero. What is it?

September 2023 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...