topic Re: "Splunk must be restarted" message will not go away. in Deployment Architecture

"Splunk must be restarted" message will not go away.

I_am_Jeff — Fri, 27 Jan 2012 19:33:06 GMT

Running version 4.2.3 on a dedicated indexer.

A few days ago we got the dreaded, "Splunk must be restarted for changes to take effect. Click here to restart from the Manager," message.

We restart Splunk, both via the command line and through the GUI. The message reappears.
We stop Splunk via the command line. Wait several minutes and restart. The message reappears.
We click "Clear Restart Message" and the message reappears immediately.

Any ideas? Even a hint on what logs to look at?

Re: "Splunk must be restarted" message will not go away.

I_am_Jeff — Fri, 27 Jan 2012 20:07:22 GMT

Update

I disabled S.o.S. Message appeared saying something like SoS was disabled and an index size change was made, please restart. Restarted and message has been gone for the last 30 minutes. (I previously tried changing the size of an index via the GUI to see if it would get the message to go away after a restart.) I have another indexer with SoS enabled, but no messages about restarting on that one.

I enabled SoS and message reappears. "User 'iamjeff' triggered the 'disable' action on app 'sos', and the following objects required a restart: indexes."

SoS disabled. No message again.

Re: "Splunk must be restarted" message will not go away.

I_am_Jeff — Fri, 27 Jan 2012 20:09:00 GMT

SoS is version 2.1.0 on both indexers.

Re: "Splunk must be restarted" message will not go away.

hexx — Sat, 28 Jan 2012 17:37:40 GMT

Please check the following Splunk Answer to see if it matches the issue you are encountering and if the proposed work-around will work for you.

Re: "Splunk must be restarted" message will not go away.

tmeader — Sat, 28 Jan 2012 19:17:45 GMT

I actually opened a bug on this with Splunk (don't know if they've identified the actual issue yet) and there is a workaround available in S.o.S 2.1:

http://splunk-base.splunk.com/answers/37102/persistent-splunk-must-be-restarted-for-changes-to-take-effect-ui-banner-when-accessing-the-splunk-on-splunk-apps-home-view

Hope that helps you too.

Re: "Splunk must be restarted" message will not go away.

hexx — Sat, 28 Jan 2012 20:34:15 GMT

Indeed, the issue that @tmeader is referencing is a core Splunk bug which has been filed under reference SPL-46736. For more details, please read the Splunk Answer referenced above.

Re: "Splunk must be restarted" message will not go away.

hexx — Sat, 28 Jan 2012 20:41:35 GMT

Note that the SoS app in its version 2.x should not be installed on your indexers, only on your search-head. For best practices on deploying SoS in a distributed environment, please read this Splunk Answer.