topic Re: Is there any way to apply cluster-bundle without rolling restart and search interruption? in Getting Data In

Is there any way to apply cluster-bundle without rolling restart and search interruption?

allurirohan — Tue, 08 Aug 2017 20:09:03 GMT

I have a use case where we're updating props.conf frequently. We'd like to ideally be able to do this on an ad-hoc basis so that we don't have people waiting 'till the next day to start seeing their data parsed properly.

However, that conflicts with the people who are already using Splunk - we cannot simply go and break searches on the Search Heads while they're using them.

Is there any other way apply cluster-bundle without rolling restart and search interruption?

Re: Is there any way to apply cluster-bundle without rolling restart and search interruption?

somesoni2 — Tue, 08 Aug 2017 20:49:21 GMT

The data parsing should happen on your Indexers or heavy forwarders right?

Re: Is there any way to apply cluster-bundle without rolling restart and search interruption?

allurirohan — Tue, 08 Aug 2017 20:50:56 GMT

Yes, On indexers

Re: Is there any way to apply cluster-bundle without rolling restart and search interruption?

somesoni2 — Tue, 08 Aug 2017 21:28:44 GMT

So, you're deploying Field extractions/transforms to your SHC and don't want to restart? If yes, then you can use this method of cluster bundle deployment to push changes without initiating rolling restart.

http://docs.splunk.com/Documentation/Splunk/6.6.2/DistSearch/PropagateSHCconfigurationchanges#Control_the_restart_process

Again, some changes do require a restart. For information on which configuration changes trigger restart, see $SPLUNK_HOME/etc/system/default/app.conf. It lists the configuration files that do not trigger restart when changed. All other configuration changes trigger restart.

Re: Is there any way to apply cluster-bundle without rolling restart and search interruption?

maciep — Tue, 08 Aug 2017 23:25:54 GMT

If you're referring to pushing parsing changes to the indexer cluster, then you can review this documentation to understand what will cause a restart vs a reload

Restart or Reload

Also, in 6.6 they finally added the ability to check whether the cluster will restart or not. So you should at least be able to determine whether one will be needed - if not, then you might be able to make the changes during business hours. We're not on 6.6 so I haven't had a chance to verify it works as expected.

Validate the Bundle

To validate the bundle and check whether a restart is necessary, include the --check-restart parameter:

splunk validate cluster-bundle --check-restart
This version of the command first validates the bundle. If validation succeeds, it then checks whether a peer restart is necessary.

If you're on an earlier version of Splunk, be sure to choose your version in the documentation. Changes have become less impactful in recent versions, but older versions will probably require a restart more often.

Re: Is there any way to apply cluster-bundle without rolling restart and search interruption?

FritzWittwer_ol — Wed, 09 Aug 2017 06:46:50 GMT

We have implemented an architecture with heavy forwarders in front of the indexer cluster, no data source connects directly to the indexers, the parsing is done on the heavy forwarders. So for most changes only the heavy forwarders are affected and restarted one by one, which saves us a rolling restart of the cluster.

Re: Is there any way to apply cluster-bundle without rolling restart and search interruption?

allurirohan — Wed, 09 Aug 2017 14:34:33 GMT

we have the configuration where some of the parsing is done at index time vs search time (Reports etc), how will that be handled with your architecture? when we create the rules from GUI which will not force the rolling restart right? correct me if I am wrong. Can we use the same RESTAPI that GUI invokes from backend servers?

Re: Is there any way to apply cluster-bundle without rolling restart and search interruption?

FritzWittwer_ol — Thu, 10 Aug 2017 05:59:01 GMT

We separated all our configurations into different apps:

UI-Apps for the Search Heads
IA-Apps for the Indexer
FA-Apps for the Heavy and Universal Forwarders.
and some more for special cases

All this apps are maintained in a git repository and distributed from there. Theses apps are developed and maintained on separate integration servers or local test installation. The finished apps are then, as a directory tree, into the git repository. The deployment from the git repository to the deployment server, the deployer and the master node is currently a manual task.

At the moment, we are working on a continuous integration / continuous delivery solution where we will control the workflows with something like Jenkins or Bamboo. The goal is to have a workflow in which a developer commits the apps to the git repo which then triggers a check with Splunk Appinspect, and a deployment to the integration servers. An accepted change could then be merged into another git branch which in turn would trigger deployment to the production environment. A deployment which results in a rolling restart will be queued to some defined 'quiet' hours.

Of course, this description is a bit simplified as reality is always a bit more complicated, but it should explain the big picture.

Re: Is there any way to apply cluster-bundle without rolling restart and search interruption?

allurirohan — Thu, 10 Aug 2017 14:38:36 GMT

ok..Thanks !