topic Re: Intermittent Unclean Shutdowns in Splunk Search

Intermittent Unclean Shutdowns

tgiles — Wed, 06 Apr 2011 22:34:53 GMT

Hi, All.

I'm setting up a new indexer and have run into an intermittent issue with Splunk (on 64 bit Linux) reporting an unclean shutdown and requiring to clean up before starting. It's happened the past 3 of the last 5 restarts I've done.

System is currently just reading its own system logs into a separate index from main.

(one shutdown)

bucket=/opt/splunk/var/lib/splunk/audit/db/hot_v1_0 count mismatch tsidx=1401 source-metadata=1398, repairing...
bucket=/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_0 count mismatch tsidx=4021 source-metadata=3991, repairing...

(another shutdown)

bucket=/opt/splunk/var/lib/splunk/audit/db/hot_v1_1 count mismatch tsidx=18 source-metadata=15, repairing...
bucket=/opt/splunk/var/lib/splunk/blockSignature/db/hot_v1_1 count mismatch tsidx=77 source-metadata=72, repairing...
bucket=/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_1 count mismatch tsidx=1800 source-metadata=1701, repairing...

Questions I have is:

Is this to be expected in 4.2? I had 4.1 on some testing devices and never run into a problem with unclean shutdowns, even though it was restarted dozens of times.
When it detects an unclean shutdown, /etc/init.d/splunk is useless- the system requires manual intervention. Is there a way to have Splunk automatically recover when it detects an unclean shutdown?

Thanks in advance for your input!

Re: Intermittent Unclean Shutdowns

tgiles — Sat, 09 Apr 2011 04:31:32 GMT

As a quick update- performing a full memory test on the system over the weekend to see if that might pin down the problem. Will update once I know more. Cheers

Re: Intermittent Unclean Shutdowns

tgiles — Mon, 11 Apr 2011 21:03:24 GMT

No memory or hard drive issues with the target system. I tested a second indexer, restarted splunk a few times, and it came up with a "Splunk has detected an unclean shutdown." message as well.

Note, this is different from SPL-37510. I'm not restarting it, going from a cold stop.

Re: Intermittent Unclean Shutdowns

tgiles — Tue, 12 Apr 2011 03:02:35 GMT

After removing the "blockSignSize = 100" setting from the index, alerts on restart disappeared completely.

Unsure if that's the right answer, but it seems to handle my issue at this time. Will continue to research and look for any future alerts from the system.

Re: Intermittent Unclean Shutdowns

southeringtonp — Wed, 04 May 2011 18:11:55 GMT

Did this resolve your issue in the long term, and/or did you ever get any additional information?