help with search head pooling + mounted knowledge bundle

[tpsplunk]

in the manual: http://docs.splunk.com/Documentation/Splunk/4.2.3/Deploy/Mounttheknowledgebundle#Use_mounted_bundles...

there is this statement as a pre-req: "You have mounted one search head's $SPLUNK_HOME/etc/system directory to the same shared storage location that the pool is using."

but there are no instructions that i can find that explain exactly what that means. How do i do this? Why mount only one search heads $SPLUNK_HOME/etc/system dir?

when you follow the instructions for search head pooling you add an NFS mount to some shared storage, copy the contents of $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users to the NFS mount and then configure splunk via server.conf to look at the NFS dirs. Does the $SPLUNK_HOME/etc/system work differently?

[ewoo · Accepted solution]

The idea here is that since all the Splunk instances in a pool should be on the same version, they all have identical default configurations in $SPLUNK_HOME/etc/system. As a result, all indexers can share a single search head's system defaults.

$SPLUNK_HOME/etc/system does work differently from $SPLUNK_HOME/etc/{apps,users} in that each instance in a pool uses its own local system directory even when its {apps,users} directories are diverted to shared storage. This is because each instance needs to have the ability to maintain some custom state independent of the other instances, e.g. different splunkweb/splunkd ports. $SPLUNK_HOME/etc/system is reserved for this purpose.

This is why especially with a search head pool with mounted bundles, it is best to place all search knowledge (props, macros, event types, etc.) in apps, not in system.

[ewoo]

Correct, indexers will use the system directory on shared storage, but search heads will ignore it, continuing to use their own local versions.

To "mount" the system dir, simply copy it from one of the search heads over to shared storage.

[tpsplunk]

so indexers will use the shared system, but searcheads ignore it?
I think the guide is a bit vague on exactly what you are supposed to do to "mount" the system dir to shared storage from a single search head. is this a correct interpretation? presume /mnt/shp is shared storage on all indexers and searchheads:
from one search head create a symlink from the shared disk to local system: ln -s $SPLUNK_HOME/etc/system /mnt/shp/search01-system
create a symlink to that link: ln -s /mnt/shp/etc/system /mnt/shp/search01-system
configure the indexers to look at /mnt/shp/etc for {apps,users,system}