Is it recommended to use Mounted Bundles in Search Head Clustering?

[rbal_splunk]

We have very large deployment of Splunk and have been using Search lead Pooling along with Mounted bundle in 6.x (Pre- 6.2), Now we are planning to move away from Search head Pooling to Search Head Cluster. What is the recommendation? Do we continue to use Mounted Bundle?

[rbal_splunk · Accepted solution]

Before we go into the detail of how the mounted bundle with search head clustering, For most types of deployments, including search head clustering, Splunk recommends that you use normal bundle replication, rather than mounted bundles with shared storage. From our docs -- http://docs.splunk.com/Documentation/Splunk/6.2.5/DistSearch/Migratefromsearchheadpooling#Search_hea...

As a result of changes to bundle replication made in the 5.0 timeframe, such as the introduction of delta-based replication and improvements in streaming, the practical use case for mounted bundles is now extremely limited. In most cases, mounted bundles make little difference in the amount of network traffic or the speed at which bundle changes get distributed to the search peers. At the same time, they add significant management complexity, particularly when combined with shared storage. Because of delta-based replication, even if your configurations contain large files, normal bundle replication entails little ongoing replication cost, as long as those files rarely change.

Before considering Mounted Bundle recommendation will be to try regular bundle replication , to see if the improvements made since 5.0 (e.g. delta-based replication) solve their problems?

Some of the customer have migrate off mounted bundles during their transition from search head pooling to search head clustering.

But if due to some reason you need to stick with mounted bundles, you will need probably need to update distsearch.conf on the indexer tier to refer to the search head cluster GUID instead of serverName.

[searchhead:]
mounted_bundles = true
bundles_location =

Before search head clustering the mounted bundle definition require Server Name like in the above example.

Mounted bundles in Search Head Clustering require cluster GUID instead of server name in distsearch.conf on the indexer. The GUID for the Search Head Cluster can be running shcluster-status as shown below.

$ splunk show shcluster-status -auth
admin:changeme Captain:
dynamic_captain : 1
elected_captain : Fri Aug 21 20:18:30
2015
id :
C7729EE6-D260-4268-A699-C1F95AAD07D5 <<< SHCLUSTER GUID!
initialized_flag : 0
label : xxx-xxxxx
maintenance_mode : 0
mgmt_uri : https://localhost:2001
min_peers_joined_flag : 0
rolling_restart_flag : 0
service_ready_flag : 0

NOTE : The GUID for Search head Cluster member is “conf id = 63A37C0A-1986-4AAD-8235-D19A62B41C7A”, so the clarify the stanza in distsearch.conf on the indexer wil ook like

To clarify, I'm referring to stanzas like:

[searchhead:]
mounted_bundles=true
bundles_location=

Using above example using Search Head Cluster GUID stanza will be like

[searchhead:C7729EE6-D260-4268-A699-C1F95AAD07D5]
mounted_bundles=true
bundles_location=

Described here: http://docs.splunk.com/Documentation/Splunk/6.2.4/DistSearch/Configuremountedbundles#Configure_the_s...

So while Migration to search head clustering is usually a great opportunity to move customers back to regular bundle replication. In general, recommendation is to ONLY use mounted bundles with search head clustering unless it is truly, deeply necessary.
Use of regular bundle replication should be acceptable in the majority of cases. The one notable exception is: large, frequently-updated lookup tables.

Note: Large, infrequently-updated lookup tables should work reasonably well in 5.0 and higher (delta replication).

If due to some reason you have to use Mounted Bundle with Search Head Cluster here is how the working on the Bundle

Since all of the Search head Cluster Members "check-in" any search time change to a captain, only captain will send knowledge bundle to peers(indexers) with a shared id defined in [shclustering] in server.conf.

1. All peers store the shared knowledge bundle
2. All search heads use the same id to share the knowledge bundle stored in peers.
3. A search -> Checks peers for the latest available bundle -> Request map-search to peers.