Deployment Architecture

Splunk Web does not restart with bucket restore

Arkon
Explorer

Hello,

When restoring buckets to thawedb, Splunk Web does not want to restart.

Here is the procedure:

  • A Warm Bucket containing the main index is stored on a backup server
  • I download the bucket into /tmp/
  • I chown all the files and directories of the bucket with splunk:splunk
  • Splunk fsck on the bucket directory
  • Move the bucket to thawedb
  • create meta.dirty
  • restart splunk

Here are the main lines of the crash log file:

Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 31817 running under UID 497.
 Crashing thread: SplunkdSpecificInitThread

 Backtrace:
  [0x00007F2E41B385F7] gsignal + 55 (/lib64/libc.so.6)
  [0x00007F2E41B39CE8] abort + 328 (/lib64/libc.so.6)
  [0x00007F2E41B31566] ? (/lib64/libc.so.6)
  [0x00007F2E41B31612] ? (/lib64/libc.so.6)
  [0x0000000000BBE5E7] _ZN14IndexerService35disableIndexesAndReinitGlobalConfigERKN9__gnu_cxx17__normal_iteratorIPK3StrSt6vectorIS2_SaIS2_EEEESA_ + 503 (splunkd)
  [0x0000000000BBF005] _ZN14IndexerService18initPerIndexConfigEP9StrVectorb + 309 (splunkd)
  [0x0000000000BC042C] _ZN14IndexerService12reloadConfigERK14IndexConfigRef + 428 (splunkd)
  [0x0000000001011BF3] _ZN9EventLoop20internal_runInThreadEP13InThreadActorb + 291 (splunkd)
  [0x0000000000BBD9BB] _ZN14IndexerService16loadLatestConfigEP14IndexConfigRef + 411 (splunkd)
  [0x0000000000BBDB65] _ZN14IndexerService16loadLatestConfigEv + 21 (splunkd)
  [0x0000000000BBDEA2] _ZN14IndexerServiceC2Ev + 786 (splunkd)
  [0x0000000000BBE231] _ZN14IndexerService14_new_singletonEv + 65 (splunkd)
  [0x00000000009AF647] _ZN25SplunkdSpecificInitThread4mainEv + 135 (splunkd)
  [0x00000000010A423E] _ZN6Thread8callMainEPv + 62 (splunkd)
  [0x00007F2E41ECCDC5] ? (/lib64/libpthread.so.0)
  [0x00007F2E41BF9C9D] clone + 109 (/lib64/libc.so.6)
 Linux / splunk.myhost.local / 4.4.10-22.54.amzn1.x86_64 / #1 SMP Tue May 17 22:45:04 UTC 2016 / x86_64
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2016-05-24 14:47:05.530 -0700 splunkd started (build cae2458f4aef)
    splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.
    2016-05-24 14:52:51.066 -0700 splunkd started (build cae2458f4aef)
    2016-05-24 14:54:51.823 -0700 Interrupt signal received
    2016-05-24 14:55:12.773 -0700 splunkd started (build cae2458f4aef)
    splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.

Thread: "SplunkdSpecificInitThread", did_join=0, ready_to_run=Y, main_thread=N

What am I missing here?
Thank you very much in advance

0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

What is shown in splunkd.log?

View solution in original post

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee

Firstly you may need to find the reason for the below message and why it tries to disable the default index. Most common cases are bucket id conflict. Check the splunkd.log

splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

What is shown in splunkd.log?

0 Karma

Arkon
Explorer

Bucket ID Collision! Thanks!

0 Karma

jkat54
SplunkTrust
SplunkTrust

Does your indexes.conf have the default index disabled?

disabled=1

??

0 Karma

Arkon
Explorer

No it is not

0 Karma
*NEW* Splunk Love Promo!
Snag a $25 Visa Gift Card for Giving Your Review!

It's another Splunk Love Special! For a limited time, you can review one of our select Splunk products through Gartner Peer Insights and receive a $25 Visa gift card!

Review:





Or Learn More in Our Blog >>