I'm setting up a new indexer and have run into an intermittent issue with Splunk (on 64 bit Linux) reporting an unclean shutdown and requiring to clean up before starting. It's happened the past 3 of the last 5 restarts I've done.
System is currently just reading its own system logs into a separate index from main.
bucket=/opt/splunk/var/lib/splunk/audit/db/hot_v1_0 count mismatch tsidx=1401 source-metadata=1398, repairing... bucket=/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_0 count mismatch tsidx=4021 source-metadata=3991, repairing...
bucket=/opt/splunk/var/lib/splunk/audit/db/hot_v1_1 count mismatch tsidx=18 source-metadata=15, repairing... bucket=/opt/splunk/var/lib/splunk/blockSignature/db/hot_v1_1 count mismatch tsidx=77 source-metadata=72, repairing... bucket=/opt/splunk/var/lib/splunk/_internaldb/db/hot_v1_1 count mismatch tsidx=1800 source-metadata=1701, repairing...
Questions I have is:
Is this to be expected in 4.2? I had 4.1 on some testing devices and never run into a problem with unclean shutdowns, even though it was restarted dozens of times.
When it detects an unclean shutdown, /etc/init.d/splunk is useless- the system requires manual intervention. Is there a way to have Splunk automatically recover when it detects an unclean shutdown?
Thanks in advance for your input!
As a quick update- performing a full memory test on the system over the weekend to see if that might pin down the problem. Will update once I know more. Cheers
No memory or hard drive issues with the target system. I tested a second indexer, restarted splunk a few times, and it came up with a "Splunk has detected an unclean shutdown." message as well.
Note, this is different from SPL-37510. I'm not restarting it, going from a cold stop.
After removing the "blockSignSize = 100" setting from the index, alerts on restart disappeared completely.
Unsure if that's the right answer, but it seems to handle my issue at this time. Will continue to research and look for any future alerts from the system.