All Apps and Add-ons

TA is not working on search head cluster

SplunkerPaul
Engager

I create a TA which is working fine in our Splunk test environment. However, seems that any configuration from props.conf and transforms.conf is not applied in the production environment.  There are events in XML format that KV_MODE = xml setting in props.conf makes field extraction just fine in test however the same setup doesn't work for any field extraction in a production environment. 

Running search in verbose mode.

All TA abjects are set to global and everyone read permissions.  i.e. default.meta file 

[]
access = read : [ * ], write : [ admin ]
export = system

btool props check shows that all the objects are applied as expected.

How to troubleshoot this issue? Where to start from?

Labels (2)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Props and transforms should be installed on all indexers in addition to SHs, especially when KV_MODE is set.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Props and transforms should be installed on all indexers in addition to SHs, especially when KV_MODE is set.

---
If this reply helps you, Karma would be appreciated.
0 Karma

SplunkerPaul
Engager

@richgalloway your initial guess was correct. Search head bundle was not replicated to indexers cluster correctly because of its large size ~2GB limitations, therefore, causing fields extraction issue.

Error message that helps identify the similar issue :

03-10-2021 17:19:56.535 +0100 ERROR DistributedBundleReplicationManager - Bundle with size=2009MB, path=/opt/splunk/var/run/3CCFDCED-A2CA-4409-B39F-D17DFAFA3CA2-1615393036.bundle, is too large for replication, max_size=2000MB. Check for any large unwanted files in $SPLUNK_HOME/etc/.

 

0 Karma

SplunkerPaul
Engager

props and transforms are installed on all indexers in the cluster.

another SH that uses the same indexer cluster returns search results as expected.

I guess this issue is isolated to SH. 

How to troubleshoot props configuration ssues on SH?

0 Karma

richgalloway
SplunkTrust
SplunkTrust

So you've narrowed the scope of the problem to a configuration difference between two search heads.  Compare the two SHs to see where they are different.  Use btool.

splunk btool --debug props list
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...