Start Splunk & an unclean shutdown : How to answer "yes" automatically to the request for faster recovery?

[kronos121]

Hello all,

sometimes when I start splunk I get following message:

Splunk has detected an unclean shutdown. Recovery should be attempted in order to ensure accurate search results, but this may take a while. If you choose 'No' here, you will have the option to recover again upon restarting Splunk, however that recovery may take significantly longer.

Perform faster recovery now? [Y/n]

Is it possible to start Splunk from command line with out selecting "Perform faster recovery now? [Y/n]". I would like to automatically select Yes option when unclean shutdown is reported.

Thank you and kind regards, Marko

[hexx · Accepted solution]

As you may already know, on *nix systems the start-up of Splunk during system boot is normally controlled by the /etc/init.d/splunk script. Looking at the "start" procedure in that script, we use "--no-prompt" as the default start-up option :

splunk_start() {
  echo Starting Splunk...
  "/home/support/sas/bin/splunk" start --no-prompt
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch /var/lock/subsys/splunk
}

This is in concordance with the behavior reported here when considering the definition of the "--no-prompt" option - http://www.splunk.com/base/Documentation/latest/Installation/StartSplunkforthefirsttime#Other_start_... :

There are two other start options: no-prompt and answer-yes:
* If you run $SPLUNK_HOME/bin/splunk start --no-prompt, Splunk proceeds with startup until it requires you to answer a question. Then, it displays the question, why it is quitting, and quits.
* If you run SPLUNK_HOME/bin/splunk start --answer-yes, Splunk proceeds with startup and automatically answers "yes" to all yes/no questions. Splunk displays the question and answer as it continues.

You could add "--answer-yes" after "--no-prompt" in the start procedure of /etc/init.d/splunk which would result in the following behavior on start-up :

• Splunk answers "yes" to any "yes/no" question.
• Splunk quits when it encounters a non-"yes/no" question.

Now keep in mind that this is a change that you would have to make at your own risk. It might be preferable for the Splunk admin to be clearly aware of the existence of index consistency issues. But if you would like to change the behavior of /etc/init.d/splunk in the way we just discussed, that's how you can do it.

IMPORTANT UPDATE : As of Splunk 4.2.3, this prompt no longer occurs and the user is notified instead upon restart after an unclean shutdown that a manual check of the indexes might be a good idea :

Splunk has detected an unclean shutdown.  The database should be checked in

order to ensure correct search results, but this may take a very long time,

depending on your system.

If you would like to check/repair the database, stop Splunk and run:

  splunk fsck --all --repair

[Michael]

Bug in "splunk enable boot-start"?

I seem to experience this same thing when I restart my Linux (Redhat) systems. It appears that it's not shutting down properly -- and when it attempts to start back up, it sees an improperly closed database -- then, while it appears that the process is running, it's not until you issue a manual "splunk start" you'll see the "unclean shutdown errorr" and be prompted to fix it.

The "fix" noted above merely shows you how to do this ("./splunk start --answer-yes ") -- and gets you going, but doesn't fix the underlying issue. You can easily re-create this by simply killing the process (without a "splunk stop" then doing a "splunk start" -- this is essentially what's happening when you reboot.

In looking at the rc.d files, there's startup commands issued (/etc/rc.d/rc3.d/S90splunk) for Splunk, but no shutdown (i.e., missing: /etc/rc.d/rc3.d/K90splunk). In fact, I find a start file in rc3.d, rc4.d, and rc5.d -- but no shutdown ones.

You have to create your own "K file" to shut it down properly in the first place (/opt/splunk/bin/splunk stop).

I would think that a database system with it's own mechanism for creating startups, would also provide for a clean shutdown in init.d. I'm chalking this up to a faulty "splunk enable boot-start".

[vbumgarner]

Any way to do this on Windows?

[bfaber]

seeing the same error here. Nothing in docs.