Splunk Web does not restart with bucket restore

[Arkon]

Hello,

When restoring buckets to thawedb, Splunk Web does not want to restart.

Here is the procedure:

• A Warm Bucket containing the main index is stored on a backup server
• I download the bucket into /tmp/
• I chown all the files and directories of the bucket with splunk:splunk
• Splunk fsck on the bucket directory
• Move the bucket to thawedb
• create meta.dirty
• restart splunk

Here are the main lines of the crash log file:

Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 31817 running under UID 497.
 Crashing thread: SplunkdSpecificInitThread

 Backtrace:
  [0x00007F2E41B385F7] gsignal + 55 (/lib64/libc.so.6)
  [0x00007F2E41B39CE8] abort + 328 (/lib64/libc.so.6)
  [0x00007F2E41B31566] ? (/lib64/libc.so.6)
  [0x00007F2E41B31612] ? (/lib64/libc.so.6)
  [0x0000000000BBE5E7] _ZN14IndexerService35disableIndexesAndReinitGlobalConfigERKN9__gnu_cxx17__normal_iteratorIPK3StrSt6vectorIS2_SaIS2_EEEESA_ + 503 (splunkd)
  [0x0000000000BBF005] _ZN14IndexerService18initPerIndexConfigEP9StrVectorb + 309 (splunkd)
  [0x0000000000BC042C] _ZN14IndexerService12reloadConfigERK14IndexConfigRef + 428 (splunkd)
  [0x0000000001011BF3] _ZN9EventLoop20internal_runInThreadEP13InThreadActorb + 291 (splunkd)
  [0x0000000000BBD9BB] _ZN14IndexerService16loadLatestConfigEP14IndexConfigRef + 411 (splunkd)
  [0x0000000000BBDB65] _ZN14IndexerService16loadLatestConfigEv + 21 (splunkd)
  [0x0000000000BBDEA2] _ZN14IndexerServiceC2Ev + 786 (splunkd)
  [0x0000000000BBE231] _ZN14IndexerService14_new_singletonEv + 65 (splunkd)
  [0x00000000009AF647] _ZN25SplunkdSpecificInitThread4mainEv + 135 (splunkd)
  [0x00000000010A423E] _ZN6Thread8callMainEPv + 62 (splunkd)
  [0x00007F2E41ECCDC5] ? (/lib64/libpthread.so.0)
  [0x00007F2E41BF9C9D] clone + 109 (/lib64/libc.so.6)
 Linux / splunk.myhost.local / 4.4.10-22.54.amzn1.x86_64 / #1 SMP Tue May 17 22:45:04 UTC 2016 / x86_64
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2016-05-24 14:47:05.530 -0700 splunkd started (build cae2458f4aef)
    splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.
    2016-05-24 14:52:51.066 -0700 splunkd started (build cae2458f4aef)
    2016-05-24 14:54:51.823 -0700 Interrupt signal received
    2016-05-24 14:55:12.773 -0700 splunkd started (build cae2458f4aef)
    splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.

Thread: "SplunkdSpecificInitThread", did_join=0, ready_to_run=Y, main_thread=N

What am I missing here?
Thank you very much in advance

[sylim_splunk]

Firstly you may need to find the reason for the below message and why it tries to disable the default index. Most common cases are bucket id conflict. Check the splunkd.log

splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.

[esix_splunk · Accepted solution]

What is shown in splunkd.log?

[Arkon]

Bucket ID Collision! Thanks!

[jkat54]

Does your indexes.conf have the default index disabled?

disabled=1

??

[Arkon]

No it is not