TA is not working on search head cluster

[SplunkerPaul]

I create a TA which is working fine in our Splunk test environment. However, seems that any configuration from props.conf and transforms.conf is not applied in the production environment.  There are events in XML format that KV_MODE = xml setting in props.conf makes field extraction just fine in test however the same setup doesn't work for any field extraction in a production environment. 

Running search in verbose mode.

All TA abjects are set to global and everyone read permissions.  i.e. default.meta file 

[]
access = read : [ * ], write : [ admin ]
export = system

btool props check shows that all the objects are applied as expected.

How to troubleshoot this issue? Where to start from?

[richgalloway · Accepted solution]

Props and transforms should be installed on all indexers in addition to SHs, especially when KV_MODE is set.

[SplunkerPaul]

@richgalloway your initial guess was correct. Search head bundle was not replicated to indexers cluster correctly because of its large size ~2GB limitations, therefore, causing fields extraction issue.

Error message that helps identify the similar issue :

03-10-2021 17:19:56.535 +0100 ERROR DistributedBundleReplicationManager - Bundle with size=2009MB, path=/opt/splunk/var/run/3CCFDCED-A2CA-4409-B39F-D17DFAFA3CA2-1615393036.bundle, is too large for replication, max_size=2000MB. Check for any large unwanted files in $SPLUNK_HOME/etc/.

[SplunkerPaul]

props and transforms are installed on all indexers in the cluster.

another SH that uses the same indexer cluster returns search results as expected.

I guess this issue is isolated to SH. 

How to troubleshoot props configuration ssues on SH?

[richgalloway]

So you've narrowed the scope of the problem to a configuration difference between two search heads.  Compare the two SHs to see where they are different.  Use btool.

splunk btool --debug props list