topic Re: Transforms are working on a local Splunk instance, but why are fields not extracted correctly when deployed to my search head cluster? in Splunk Search

Transforms are working on a local Splunk instance, but why are fields not extracted correctly when deployed to my search head cluster?

paimonsoror — Tue, 29 Sep 2020 11:46:09 GMT

Having a strange issue. I am trying to set up a transform to automatically extract key/value pairs from a non standard XML file. I have a local Splunk instance where I did this development. Here is what I did:

Did a Add New of a data source
Selected a local copy of the XML file
The data was read once (not set to monitor the file)
Data was indexed into an index called 'app_test'
Data is in the Search app

I then went ahead and updated my props.conf and transforms.conf in my $SPLUNK_ROOT/etc/apps/search/local

props.conf

[dmwt:xml]
REPORT-myXmlClassName = xmlTransform

transforms.conf

[xmlTransform]
REGEX = \<(\w+[^\n\/\>]+)\/?\>([^\<\n][^\<]*)\<
FORMAT = $1::$2

The extraction worked beautifully. So next I went to go add it to my clustered environment. I have the XML data being indexed to an index called app_dmwt. I also created an app called 'DMWT'. I created similar additions to the props.conf and transforms.conf, but this time put it in $SPLUNK_ROOT/etc/shcluster/app/DMWT/default . Note that the difference here is that instead of doing an add datasource from the UI, I have a forwarder monitoring 3 files each as their own sourcetype.

props.conf

[ dmwt:delete ]
REPORT-xmlkv = xmlkv-alternative

[ dmwt:insert ]
REPORT-xmlkv = xmlkv-alternative

[ dmwt:update ]
REPORT-xmlkv = xmlkv-alternative

transforms.conf

[xmlkv-alternative]
REGEX = \<(\w+[^\n\/\>]+)\/?\>([^\<\n][^\<]*)\<
FORMAT = $1::$2

I then did a bundle push from the deployer, but for some reason, the transform isn't taking effect.

Any thoughts to what might be going on?

Re: Transforms are working on a local Splunk instance, but why are fields not extracted correctly when deployed to my search head cluster?

paimonsoror — Wed, 16 Nov 2016 14:22:00 GMT

I may have found my problem. I was applying my props on the search head cluster, not on the indexer. Correcting that now to see if it resolves the problem!

Re: Transforms are working on a local Splunk instance, but why are fields not extracted correctly when deployed to my search head cluster?

skalliger — Wed, 16 Nov 2016 15:19:54 GMT

Yea, the transforming is done on the indexers, not the search heads.
But I am also wondering why you tried to deploy the changes in the "default" directory. This is only done when you deploy the first time. When you update the configuration, you usually store it under \local\ and don't change \default\ files..

Skalli

Re: Transforms are working on a local Splunk instance, but why are fields not extracted correctly when deployed to my search head cluster?

Afef — Wed, 16 Nov 2016 15:20:14 GMT

did you solve the problem ?

Re: Transforms are working on a local Splunk instance, but why are fields not extracted correctly when deployed to my search head cluster?

paimonsoror — Wed, 16 Nov 2016 16:02:31 GMT

Interestingly enough, it didn't even work on the indexers when i pushed the bundle. I was messing around and came back to pushing them to the forwarders. Once I did an

index=whatever | extract reload=t

it updated everything....

Also, the reason i am working in default is because i am developing this app in a lower environment for my users to test.