Splunk Enterprise

App Modifications on Search Head Cluster

Path Finder

My app resides on all search head cluster. I need to modify default files and place in local directory and push across all search heads. I'm placing my app on shcluster folder on deployment server and issued "splunk apply shcluster-bundle -target ". But new files were not placed into the local directory. Can anyone let me know where I need to place the files to update across all the search heads.

Tags (1)
0 Karma

Super Champion

SH clustering is bit complex (especially if you have Enterprise Security)
The three elements to do properly a SH cluster are

  1. Staging Server - this is a standalone system whereby you produce all the code and make your changes
  2. Deployer - This can be same as Staging SErver, but the location of the apps should be in "etc/shcluster/apps" and will push to all sh-members
  3. SH members - The final location of apps

So when you create a new app yourself, I tend to put the code in "local". So the app you create , you create in "Staging Server". Pass to deployer and deploy to SH-members. But when it comes to SH-members, it will appear into "default" , no matter you make the changes in "local" in your staging-server. (This is a pain for version-control though). Ensure you pass "-preserve_lookups true" while deploying from deployer, so that the lookups in the SH-members are not overwritten.

So SH-cluster flow in large environments is for your myapp would be:

Staging Server ($SPLUNK_HOME/etc/apps/myapp/) -> deployer ($SPLUNK_HOME/etc/shcluster/apps/myapp) -> SH-members ( $SPLUNK_HOME/etc/apps/myapp)

Going into the file level configurations # I agree this is BAD, but this is how Splunk works

Staging Server ($SPLUNK_HOME/etc/apps/myapp/local/server.conf ) => deployer ($SPLUNK_HOME/etc/shcluster/apps/myapp) => SH-members ($SPLUNK_HOME/etc/apps/myapp/default/server.conf

Ultra Champion

When you are "pushing" a new app, in an ideal world you don't want anything other than a 'default' folder in your app.

When an app is installed or upgraded, ONLY the default folder is changed (*). Therefore any files in the local folder on your SHC will remain as they are.

What you may wish to do is collect all the files that have been changed and are now in your SHC/local folders and merge them into the new default folder on the deployer before you push the changes.

  • () Ok, thats not quite true as all the other folders like ./bin ./lookups etc are too, but I was highlighting that specifically the ./local folder is not changed
If my comment helps, please give it a thumbs up!
0 Karma
Get Updates on the Splunk Community!

Build Scalable Security While Moving to Cloud - Guide From Clayton Homes

 Clayton Homes faced the increased challenge of strengthening their security posture as they went through ...

Mission Control | Explore the latest release of Splunk Mission Control (2.3)

We’re happy to announce the release of Mission Control 2.3 which includes several new and exciting features ...

Cloud Platform | Migrating your Splunk Cloud deployment to Python 3.7

Python 2.7, the last release of Python 2, reached End of Life back on January 1, 2020. As part of our larger ...