Deployment Architecture

Splunk Web does not restart with bucket restore

Explorer

Hello,

When restoring buckets to thawedb, Splunk Web does not want to restart.

Here is the procedure:

  • A Warm Bucket containing the main index is stored on a backup server
  • I download the bucket into /tmp/
  • I chown all the files and directories of the bucket with splunk:splunk
  • Splunk fsck on the bucket directory
  • Move the bucket to thawedb
  • create meta.dirty
  • restart splunk

Here are the main lines of the crash log file:

Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 31817 running under UID 497.
 Crashing thread: SplunkdSpecificInitThread

 Backtrace:
  [0x00007F2E41B385F7] gsignal + 55 (/lib64/libc.so.6)
  [0x00007F2E41B39CE8] abort + 328 (/lib64/libc.so.6)
  [0x00007F2E41B31566] ? (/lib64/libc.so.6)
  [0x00007F2E41B31612] ? (/lib64/libc.so.6)
  [0x0000000000BBE5E7] _ZN14IndexerService35disableIndexesAndReinitGlobalConfigERKN9__gnu_cxx17__normal_iteratorIPK3StrSt6vectorIS2_SaIS2_EEEESA_ + 503 (splunkd)
  [0x0000000000BBF005] _ZN14IndexerService18initPerIndexConfigEP9StrVectorb + 309 (splunkd)
  [0x0000000000BC042C] _ZN14IndexerService12reloadConfigERK14IndexConfigRef + 428 (splunkd)
  [0x0000000001011BF3] _ZN9EventLoop20internal_runInThreadEP13InThreadActorb + 291 (splunkd)
  [0x0000000000BBD9BB] _ZN14IndexerService16loadLatestConfigEP14IndexConfigRef + 411 (splunkd)
  [0x0000000000BBDB65] _ZN14IndexerService16loadLatestConfigEv + 21 (splunkd)
  [0x0000000000BBDEA2] _ZN14IndexerServiceC2Ev + 786 (splunkd)
  [0x0000000000BBE231] _ZN14IndexerService14_new_singletonEv + 65 (splunkd)
  [0x00000000009AF647] _ZN25SplunkdSpecificInitThread4mainEv + 135 (splunkd)
  [0x00000000010A423E] _ZN6Thread8callMainEPv + 62 (splunkd)
  [0x00007F2E41ECCDC5] ? (/lib64/libpthread.so.0)
  [0x00007F2E41BF9C9D] clone + 109 (/lib64/libc.so.6)
 Linux / splunk.myhost.local / 4.4.10-22.54.amzn1.x86_64 / #1 SMP Tue May 17 22:45:04 UTC 2016 / x86_64
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2016-05-24 14:47:05.530 -0700 splunkd started (build cae2458f4aef)
    splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.
    2016-05-24 14:52:51.066 -0700 splunkd started (build cae2458f4aef)
    2016-05-24 14:54:51.823 -0700 Interrupt signal received
    2016-05-24 14:55:12.773 -0700 splunkd started (build cae2458f4aef)
    splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.

Thread: "SplunkdSpecificInitThread", did_join=0, ready_to_run=Y, main_thread=N

What am I missing here?
Thank you very much in advance

0 Karma
1 Solution

Splunk Employee
Splunk Employee

What is shown in splunkd.log?

View solution in original post

0 Karma

Splunk Employee
Splunk Employee

Firstly you may need to find the reason for the below message and why it tries to disable the default index. Most common cases are bucket id conflict. Check the splunkd.log

splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.

0 Karma

Splunk Employee
Splunk Employee

What is shown in splunkd.log?

View solution in original post

0 Karma

Explorer

Bucket ID Collision! Thanks!

0 Karma

SplunkTrust
SplunkTrust

Does your indexes.conf have the default index disabled?

disabled=1

??

0 Karma

Explorer

No it is not

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!