Splunk Search

help with search head pooling + mounted knowledge bundle

tpsplunk
Communicator

in the manual: http://docs.splunk.com/Documentation/Splunk/4.2.3/Deploy/Mounttheknowledgebundle#Use_mounted_bundles...

there is this statement as a pre-req: "You have mounted one search head's $SPLUNK_HOME/etc/system directory to the same shared storage location that the pool is using."

but there are no instructions that i can find that explain exactly what that means. How do i do this? Why mount only one search heads $SPLUNK_HOME/etc/system dir?

when you follow the instructions for search head pooling you add an NFS mount to some shared storage, copy the contents of $SPLUNK_HOME/etc/apps and $SPLUNK_HOME/etc/users to the NFS mount and then configure splunk via server.conf to look at the NFS dirs. Does the $SPLUNK_HOME/etc/system work differently?

1 Solution

ewoo
Splunk Employee
Splunk Employee

The idea here is that since all the Splunk instances in a pool should be on the same version, they all have identical default configurations in $SPLUNK_HOME/etc/system. As a result, all indexers can share a single search head's system defaults.

$SPLUNK_HOME/etc/system does work differently from $SPLUNK_HOME/etc/{apps,users} in that each instance in a pool uses its own local system directory even when its {apps,users} directories are diverted to shared storage. This is because each instance needs to have the ability to maintain some custom state independent of the other instances, e.g. different splunkweb/splunkd ports. $SPLUNK_HOME/etc/system is reserved for this purpose.

This is why especially with a search head pool with mounted bundles, it is best to place all search knowledge (props, macros, event types, etc.) in apps, not in system.

View solution in original post

ewoo
Splunk Employee
Splunk Employee

The idea here is that since all the Splunk instances in a pool should be on the same version, they all have identical default configurations in $SPLUNK_HOME/etc/system. As a result, all indexers can share a single search head's system defaults.

$SPLUNK_HOME/etc/system does work differently from $SPLUNK_HOME/etc/{apps,users} in that each instance in a pool uses its own local system directory even when its {apps,users} directories are diverted to shared storage. This is because each instance needs to have the ability to maintain some custom state independent of the other instances, e.g. different splunkweb/splunkd ports. $SPLUNK_HOME/etc/system is reserved for this purpose.

This is why especially with a search head pool with mounted bundles, it is best to place all search knowledge (props, macros, event types, etc.) in apps, not in system.

ewoo
Splunk Employee
Splunk Employee

Correct, indexers will use the system directory on shared storage, but search heads will ignore it, continuing to use their own local versions.

To "mount" the system dir, simply copy it from one of the search heads over to shared storage.

0 Karma

tpsplunk
Communicator

so indexers will use the shared system, but searcheads ignore it?
I think the guide is a bit vague on exactly what you are supposed to do to "mount" the system dir to shared storage from a single search head. is this a correct interpretation? presume /mnt/shp is shared storage on all indexers and searchheads:
from one search head create a symlink from the shared disk to local system: ln -s $SPLUNK_HOME/etc/system /mnt/shp/search01-system
create a symlink to that link: ln -s /mnt/shp/etc/system /mnt/shp/search01-system
configure the indexers to look at /mnt/shp/etc for {apps,users,system}

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...