splunkd is crashing and I am getting the error message in the crash file

[kamalbeg]

Starting splunk server daemon (splunkd)...
Done
[ OK ]

Waiting for web server at https://127.0.0.1:8000 to be available.splunkd 8595 was not running.
Stopping splunk helpers...
[ OK ]
Done.
Stopped helpers.
Removing stale pid file... done.

WARNING: web interface does not seem to be available!

opt/splunk/var/log/splunk$ more crash-2018-11-20-19:21:02.log
[build 586c3ec08cfb] 2018-11-20 19:21:02
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 8595 running under UID 31964.
Crashing thread: IdataDO_Collector
Registers:
RIP: [0x00007FD030389495] gsignal + 53 (libc.so.6 + 0x32495)
RDI: [0x0000000000002193]
RSI: [0x00000000000021EC]
RBP: [0x000055D4025F1710]
RSP: [0x00007FD025BFE618]
RAX: [0x0000000000000000]
RBX: [0x00007FD0318F6000]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x0000000000000008]
R9: [0x00007FD031947598]
R10: [0x0000000000000008]
R11: [0x0000000000000206]
R12: [0x000055D4025800B9]
R13: [0x000055D4026BCA80]
R14: [0x000055D402CD0EA0]
R15: [0x00007FD024533720]
EFL: [0x0000000000000206]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x00007FD030389495] gsignal + 53 (libc.so.6 + 0x32495)
[0x00007FD03038AC75] abort + 373 (libc.so.6 + 0x33C75)
[0x00007FD03038260E] ? (libc.so.6 + 0x2B60E)
[0x00007FD0303826D0] assert_perror_fail + 0 (libc.so.6 + 0x2B6D0)
[0x000055D40139783C] ? (splunkd + 0x8CA83C)
[0x000055D40139CA7D] _ZN22IdataCollectorCallback4tickEv + 157 (splunkd + 0x8CFA7D)
[0x000055D40118FE98] _ZN17IdataDO_Collector4mainEv + 136 (splunkd + 0x6C2E98)
[0x000055D401B45F40] _ZN6Thread8callMainEPv + 64 (splunkd + 0x1078F40)
[0x00007FD0306F2AA1] ? (libpthread.so.0 + 0x7AA1)
[0x00007FD03043FBDD] clone + 109 (libc.so.6 + 0xE8BDD)
Linux / cwb02qsplunkidx03.keybank.com / 2.6.32-754.3.5.el6.x86_64 / #1 SMP Thu Aug 9 11:56:22 EDT 2
018 / x86_64
Last few lines of stderr (may contain info on assertion failure, but also could be old):
splunkd: /home/build/build-src/ivory/src/pipeline/indexer/IdataDO_Collector.cpp:372: void collec
tindexes(): Assertion ! name.empty()' failed.
2018-11-20 18:54:30.652 -0500 splunkd started (build 586c3ec08cfb)
splunkd: /home/build/build-src/ivory/src/pipeline/indexer/IdataDO_Collector.cpp:372: void collec
t__indexes(): Assertion! name.empty()' failed.
2018-11-20 19:08:48.364 -0500 splunkd started (build 586c3ec08cfb)
splunkd: /home/build/build-src/ivory/src/pipeline/indexer/IdataDO_Collector.cpp:372: void collec
t_indexes(): Assertion `! name.empty()' failed.
2018-11-20 19:21:01.445 -0500 splunkd started (build 586c3ec08cfb)
splunkd: /home/build/build-src/ivory/src/pipeline/indexer/IdataDO_Collector.cpp:372: void collec
t_indexes(): Assertion `! name.empty()' failed.

/etc/redhat-release: Red Hat Enterprise Linux Server release 6.10 (Santiago)
glibc version: 2.12
glibc release: stable
Last errno: 0
Threads running: 40
Runtime: 1.137312s
argv: [splunkd -p 8089 restart]
Regex JIT disabled due to SELinux

Thread: "IdataDO_Collector", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7fd02a414f10:
00000000 00 f7 bf 25 d0 7f 00 00 |...%....|
00000008

x86 CPUID registers:
0: 0000000D 756E6547 6C65746E 49656E69
1: 00050654 0E010800 FEFA3203 0FABFBFF
2: 76036301 00F0B5FF 00000000 00C30000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000000 00000000 00000000 00000000
6: 00000004 00000000 00000000 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300401 0000007F 00000000 00000000
B: 00000000 00000000 000000CD 0000000E
C: 00000000 00000000 00000000 00000000
😧 00000000 00000000 00000000 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000101 2C100800
80000002: 65746E49 2952286C 6F655820 2952286E
80000003: 6C6F4720 31362064 43203034 40205550
80000004: 332E3220 7A484730 00000000 00000000
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 00003028 00000000 00000000 00000000
terminating...

[kamalbeg]

Neither is required. The issue was an incorrect stanza in indexes.conf file

[]
homePath = volume:primary//db
coldPath = volume:primary//colddb
thawedPath = $SPLUNK_DB//thaweddb

frozenTimePeriodInSecs = 2592000

maxTotalDataSizeMB = 100000
disabled = false

This caused the issue for splunk indexers to crash. Once I removed the bad stanza, it worked fine.

[woodcock]

Reinstall from scratch or open a support ticket.

[ddrillic]

I agree - a support ticket!