All Apps and Add-ons

TA is not working on search head cluster

SplunkerPaul
Engager

I create a TA which is working fine in our Splunk test environment. However, seems that any configuration from props.conf and transforms.conf is not applied in the production environment.  There are events in XML format that KV_MODE = xml setting in props.conf makes field extraction just fine in test however the same setup doesn't work for any field extraction in a production environment. 

Running search in verbose mode.

All TA abjects are set to global and everyone read permissions.  i.e. default.meta file 

[]
access = read : [ * ], write : [ admin ]
export = system

btool props check shows that all the objects are applied as expected.

How to troubleshoot this issue? Where to start from?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Props and transforms should be installed on all indexers in addition to SHs, especially when KV_MODE is set.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Props and transforms should be installed on all indexers in addition to SHs, especially when KV_MODE is set.

---
If this reply helps you, Karma would be appreciated.
0 Karma

SplunkerPaul
Engager

@richgalloway your initial guess was correct. Search head bundle was not replicated to indexers cluster correctly because of its large size ~2GB limitations, therefore, causing fields extraction issue.

Error message that helps identify the similar issue :

03-10-2021 17:19:56.535 +0100 ERROR DistributedBundleReplicationManager - Bundle with size=2009MB, path=/opt/splunk/var/run/3CCFDCED-A2CA-4409-B39F-D17DFAFA3CA2-1615393036.bundle, is too large for replication, max_size=2000MB. Check for any large unwanted files in $SPLUNK_HOME/etc/.

 

0 Karma

SplunkerPaul
Engager

props and transforms are installed on all indexers in the cluster.

another SH that uses the same indexer cluster returns search results as expected.

I guess this issue is isolated to SH. 

How to troubleshoot props configuration ssues on SH?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

So you've narrowed the scope of the problem to a configuration difference between two search heads.  Compare the two SHs to see where they are different.  Use btool.

splunk btool --debug props list
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...