topic Re: Issues with Knowledge Bundle in Splunk Cluster (SHC + Index Cluster) in Splunk Search

Issues with Knowledge Bundle in Splunk Cluster (SHC + Index Cluster)

gfuente — Tue, 03 Feb 2015 15:48:48 GMT

Hello all,

We have this Splunk 6.2.1 Architecture, on Linux VM machines:

3 SH in SHC
1 Master + Deployer
3 Cluster Peers

We have an app in the SHs, that contains a big lookup (200MB) that needs to be replicated to the 3 IDXs (for filtering purposes). It seems that we are having issues with the replication of the Knowledge Bundle, as we are getting this error on the SHs (while running a query):

[indexer1name] Search Process did not exit cleanly, exit_code=255, description="exited with code 255". Please look in the search.log for this peer in the Job Inspector for more info.

And the same message for the other 2 indexers

So, i would like to know: Is the Mounted Knowledge bundle supported with SHC? (didn't found anything related in the docs: http://docs.splunk.com/Documentation/Splunk/latest/DistSearch/Mounttheknowledgebundle)

What other options do we have, as we need to replicate this lookup into the IDXs?

Thanks

Re: Issues with Knowledge Bundle in Splunk Cluster (SHC + Index Cluster)

ewoo — Tue, 03 Feb 2015 17:46:26 GMT

Mounted bundles introduce their own maintainence costs, especially in terms of understanding the performance requirements on the NFS server as search concurrency increases and the number of indexers grows.

Do you know why/how bundle replication is failing? What ERRORs/WARNs do you see on the search head in splunkd.log and on the indexers in splunkd.log/splunkd_access.log?

If it's not possible to make bundle replication work (e.g. due to network usage constraints), one other option is to blacklist the large lookup (via distsearch.conf) and then perform the lookup locally on the search head ( with "| lookup local=true").

Re: Issues with Knowledge Bundle in Splunk Cluster (SHC + Index Cluster)

theunf — Sun, 07 Jun 2015 07:34:30 GMT

I have the mounted bundle scenario working fine with :
7 SHs
1 Deployer
1 Master
8 Indexers

My knowledge bundle is also big in the number of apps, lookups and so on. Sometimes error also 255 appeared.

Used mounted bundles since SHP doing a rsync from the master, where the NFS exports was being shared, to all indexers.
With SHC it changed to a script that runs in all SHs :
1st the script checks who is the captain (splunk show shcluster-status)
if the captain is the SH running the script, it´ll rsync all non splunk default app to all indexers

Indexers distributedsearch.conf are the same from SHP .

Take a look at my question about deployer shcluser apps sync :
http://answers.splunk.com/answers/241549/how-to-prevent-deployer-from-pushing-old-content-w.html