Deployment Architecture

Splunk Web does not restart with bucket restore

Arkon
Explorer

Hello,

When restoring buckets to thawedb, Splunk Web does not want to restart.

Here is the procedure:

  • A Warm Bucket containing the main index is stored on a backup server
  • I download the bucket into /tmp/
  • I chown all the files and directories of the bucket with splunk:splunk
  • Splunk fsck on the bucket directory
  • Move the bucket to thawedb
  • create meta.dirty
  • restart splunk

Here are the main lines of the crash log file:

Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 31817 running under UID 497.
 Crashing thread: SplunkdSpecificInitThread

 Backtrace:
  [0x00007F2E41B385F7] gsignal + 55 (/lib64/libc.so.6)
  [0x00007F2E41B39CE8] abort + 328 (/lib64/libc.so.6)
  [0x00007F2E41B31566] ? (/lib64/libc.so.6)
  [0x00007F2E41B31612] ? (/lib64/libc.so.6)
  [0x0000000000BBE5E7] _ZN14IndexerService35disableIndexesAndReinitGlobalConfigERKN9__gnu_cxx17__normal_iteratorIPK3StrSt6vectorIS2_SaIS2_EEEESA_ + 503 (splunkd)
  [0x0000000000BBF005] _ZN14IndexerService18initPerIndexConfigEP9StrVectorb + 309 (splunkd)
  [0x0000000000BC042C] _ZN14IndexerService12reloadConfigERK14IndexConfigRef + 428 (splunkd)
  [0x0000000001011BF3] _ZN9EventLoop20internal_runInThreadEP13InThreadActorb + 291 (splunkd)
  [0x0000000000BBD9BB] _ZN14IndexerService16loadLatestConfigEP14IndexConfigRef + 411 (splunkd)
  [0x0000000000BBDB65] _ZN14IndexerService16loadLatestConfigEv + 21 (splunkd)
  [0x0000000000BBDEA2] _ZN14IndexerServiceC2Ev + 786 (splunkd)
  [0x0000000000BBE231] _ZN14IndexerService14_new_singletonEv + 65 (splunkd)
  [0x00000000009AF647] _ZN25SplunkdSpecificInitThread4mainEv + 135 (splunkd)
  [0x00000000010A423E] _ZN6Thread8callMainEPv + 62 (splunkd)
  [0x00007F2E41ECCDC5] ? (/lib64/libpthread.so.0)
  [0x00007F2E41BF9C9D] clone + 109 (/lib64/libc.so.6)
 Linux / splunk.myhost.local / 4.4.10-22.54.amzn1.x86_64 / #1 SMP Tue May 17 22:45:04 UTC 2016 / x86_64
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2016-05-24 14:47:05.530 -0700 splunkd started (build cae2458f4aef)
    splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.
    2016-05-24 14:52:51.066 -0700 splunkd started (build cae2458f4aef)
    2016-05-24 14:54:51.823 -0700 Interrupt signal received
    2016-05-24 14:55:12.773 -0700 splunkd started (build cae2458f4aef)
    splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.

Thread: "SplunkdSpecificInitThread", did_join=0, ready_to_run=Y, main_thread=N

What am I missing here?
Thank you very much in advance

0 Karma
1 Solution

esix_splunk
Splunk Employee
Splunk Employee

What is shown in splunkd.log?

View solution in original post

0 Karma

sylim_splunk
Splunk Employee
Splunk Employee

Firstly you may need to find the reason for the below message and why it tries to disable the default index. Most common cases are bucket id conflict. Check the splunkd.log

splunkd: /home/build/build-src/ember/src/pipeline/indexer/IndexerService.cpp:928: void IndexerService::disableIndexesAndReinitGlobalConfig(const const_iterator&, const const_iterator&): Assertion `0 && "Cannot disable the default index."' failed.

0 Karma

esix_splunk
Splunk Employee
Splunk Employee

What is shown in splunkd.log?

0 Karma

Arkon
Explorer

Bucket ID Collision! Thanks!

0 Karma

jkat54
SplunkTrust
SplunkTrust

Does your indexes.conf have the default index disabled?

disabled=1

??

0 Karma

Arkon
Explorer

No it is not

0 Karma
Get Updates on the Splunk Community!

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Avoid Certificate Expiry Issues in Splunk Enterprise with Certificate Assist

This blog post is part 2 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Using Machine Learning for Hunting Security Threats

REGISTER NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more ...