All Apps and Add-ons

TA is not working on search head cluster

SplunkerPaul
Engager

I create a TA which is working fine in our Splunk test environment. However, seems that any configuration from props.conf and transforms.conf is not applied in the production environment.  There are events in XML format that KV_MODE = xml setting in props.conf makes field extraction just fine in test however the same setup doesn't work for any field extraction in a production environment. 

Running search in verbose mode.

All TA abjects are set to global and everyone read permissions.  i.e. default.meta file 

[]
access = read : [ * ], write : [ admin ]
export = system

btool props check shows that all the objects are applied as expected.

How to troubleshoot this issue? Where to start from?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Props and transforms should be installed on all indexers in addition to SHs, especially when KV_MODE is set.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Props and transforms should be installed on all indexers in addition to SHs, especially when KV_MODE is set.

---
If this reply helps you, an upvote would be appreciated.

View solution in original post

0 Karma

SplunkerPaul
Engager

@richgalloway your initial guess was correct. Search head bundle was not replicated to indexers cluster correctly because of its large size ~2GB limitations, therefore, causing fields extraction issue.

Error message that helps identify the similar issue :

03-10-2021 17:19:56.535 +0100 ERROR DistributedBundleReplicationManager - Bundle with size=2009MB, path=/opt/splunk/var/run/3CCFDCED-A2CA-4409-B39F-D17DFAFA3CA2-1615393036.bundle, is too large for replication, max_size=2000MB. Check for any large unwanted files in $SPLUNK_HOME/etc/.

 

0 Karma

SplunkerPaul
Engager

props and transforms are installed on all indexers in the cluster.

another SH that uses the same indexer cluster returns search results as expected.

I guess this issue is isolated to SH. 

How to troubleshoot props configuration ssues on SH?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

So you've narrowed the scope of the problem to a configuration difference between two search heads.  Compare the two SHs to see where they are different.  Use btool.

splunk btool --debug props list
---
If this reply helps you, an upvote would be appreciated.
0 Karma
.conf21 CFS Extended through 5/20!

Don't miss your chance
to share your Splunk
wisdom in-person or
virtually at .conf21!

Call for Speakers has
been extended through
Thursday, 5/20!