All Apps and Add-ons

TA is not working on search head cluster

SplunkerPaul
Engager

I create a TA which is working fine in our Splunk test environment. However, seems that any configuration from props.conf and transforms.conf is not applied in the production environment.  There are events in XML format that KV_MODE = xml setting in props.conf makes field extraction just fine in test however the same setup doesn't work for any field extraction in a production environment. 

Running search in verbose mode.

All TA abjects are set to global and everyone read permissions.  i.e. default.meta file 

[]
access = read : [ * ], write : [ admin ]
export = system

btool props check shows that all the objects are applied as expected.

How to troubleshoot this issue? Where to start from?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Props and transforms should be installed on all indexers in addition to SHs, especially when KV_MODE is set.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Props and transforms should be installed on all indexers in addition to SHs, especially when KV_MODE is set.

---
If this reply helps you, Karma would be appreciated.
0 Karma

SplunkerPaul
Engager

@richgalloway your initial guess was correct. Search head bundle was not replicated to indexers cluster correctly because of its large size ~2GB limitations, therefore, causing fields extraction issue.

Error message that helps identify the similar issue :

03-10-2021 17:19:56.535 +0100 ERROR DistributedBundleReplicationManager - Bundle with size=2009MB, path=/opt/splunk/var/run/3CCFDCED-A2CA-4409-B39F-D17DFAFA3CA2-1615393036.bundle, is too large for replication, max_size=2000MB. Check for any large unwanted files in $SPLUNK_HOME/etc/.

 

0 Karma

SplunkerPaul
Engager

props and transforms are installed on all indexers in the cluster.

another SH that uses the same indexer cluster returns search results as expected.

I guess this issue is isolated to SH. 

How to troubleshoot props configuration ssues on SH?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

So you've narrowed the scope of the problem to a configuration difference between two search heads.  Compare the two SHs to see where they are different.  Use btool.

splunk btool --debug props list
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What’s new on Splunk Lantern in August

This month’s Splunk Lantern update gives you the low-down on all of the articles we’ve published over the past ...

Welcome to the Future of Data Search & Exploration

You have more data coming at you than ever before. Over the next five years, the total amount of digital data ...

This Week's Community Digest - Splunk Community Happenings [8.3.22]

Get the latest news and updates from the Splunk Community here! News From Splunk Answers ✍️ Splunk Answers is ...