Is there any way to apply cluster-bundle without rolling restart and search interruption?

[allurirohan]

I have a use case where we're updating props.conf frequently. We'd like to ideally be able to do this on an ad-hoc basis so that we don't have people waiting 'till the next day to start seeing their data parsed properly.

However, that conflicts with the people who are already using Splunk - we cannot simply go and break searches on the Search Heads while they're using them.

Is there any other way apply cluster-bundle without rolling restart and search interruption?

[FritzWittwer_ol]

We have implemented an architecture with heavy forwarders in front of the indexer cluster, no data source connects directly to the indexers, the parsing is done on the heavy forwarders. So for most changes only the heavy forwarders are affected and restarted one by one, which saves us a rolling restart of the cluster.

[allurirohan]

we have the configuration where some of the parsing is done at index time vs search time (Reports etc), how will that be handled with your architecture? when we create the rules from GUI which will not force the rolling restart right? correct me if I am wrong. Can we use the same RESTAPI that GUI invokes from backend servers?

[FritzWittwer_ol]

We separated all our configurations into different apps:

• UI-Apps for the Search Heads
• IA-Apps for the Indexer
• FA-Apps for the Heavy and Universal Forwarders.
• and some more for special cases

All this apps are maintained in a git repository and distributed from there. Theses apps are developed and maintained on separate integration servers or local test installation. The finished apps are then, as a directory tree, into the git repository. The deployment from the git repository to the deployment server, the deployer and the master node is currently a manual task.

At the moment, we are working on a continuous integration / continuous delivery solution where we will control the workflows with something like Jenkins or Bamboo. The goal is to have a workflow in which a developer commits the apps to the git repo which then triggers a check with Splunk Appinspect, and a deployment to the integration servers. An accepted change could then be merged into another git branch which in turn would trigger deployment to the production environment. A deployment which results in a rolling restart will be queued to some defined 'quiet' hours.

Of course, this description is a bit simplified as reality is always a bit more complicated, but it should explain the big picture.

[allurirohan]

ok..Thanks !

[maciep]

If you're referring to pushing parsing changes to the indexer cluster, then you can review this documentation to understand what will cause a restart vs a reload

Restart or Reload

Also, in 6.6 they finally added the ability to check whether the cluster will restart or not. So you should at least be able to determine whether one will be needed - if not, then you might be able to make the changes during business hours. We're not on 6.6 so I haven't had a chance to verify it works as expected.

Validate the Bundle

To validate the bundle and check whether a restart is necessary, include the --check-restart parameter:

splunk validate cluster-bundle --check-restart
This version of the command first validates the bundle. If validation succeeds, it then checks whether a peer restart is necessary.

If you're on an earlier version of Splunk, be sure to choose your version in the documentation. Changes have become less impactful in recent versions, but older versions will probably require a restart more often.

[somesoni2]

The data parsing should happen on your Indexers or heavy forwarders right?

[allurirohan]

Yes, On indexers

[somesoni2]

So, you're deploying Field extractions/transforms to your SHC and don't want to restart? If yes, then you can use this method of cluster bundle deployment to push changes without initiating rolling restart.

http://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/PropagateSHCconfigurationchanges#Contro...

Again, some changes do require a restart. For information on which configuration changes trigger restart, see $SPLUNK_HOME/etc/system/default/app.conf. It lists the configuration files that do not trigger restart when changed. All other configuration changes trigger restart.