Clustered indexers constantly generating crash logs

[zijian]

Hi,

I have two clustered indexers which are now constantly generating crash logs in /splunk/var/log/splunk every few minutes and is unable to figure out the cause from the crash log or the error in splunkd.log.

Would anyone here be able to shed some light on this?

Splunkd Error:

WARN SearchProcessRunner [19356 PreforkedSearchesManager-0] - preforked process=0/38 status=killed, signum=6, signame="Aborted", coredump=1, uptime_sec=37.282768, stime_sec=19.850199,
max_rss_kb=472688, vm_minor=902282, vm_major=37, fs_r_count=608, fs_w_count=50856, sched_vol=3413, sched_invol=10923

Contents of one of the crash.log:

[build b6436b649711] 2023-11-02 11:39:40
Received fatal signal 6 (Aborted) on PID 23624.
Cause:
Signal sent by PID 23624 running under UID 1001.
Crashing thread: BucketSummaryActorThread
Registers:
RIP: [0x00007F0D7E2DA387] gsignal + 55 (libc.so.6 + 0x36387)
RDI: [0x0000000000005C48]
RSI: [0x00000000000059CC]
RBP: [0x0000000000000BE7]
RSP: [0x00007F0CF85F2268]
RAX: [0x0000000000000000]
RBX: [0x0000562A9ADF7598]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x00007F0CF85FF700]
R9: [0x00007F0D7E2F12CD]
R10: [0x0000000000000008]
R11: [0x0000000000000206]
R12: [0x0000562A9AC0E070]
R13: [0x0000562A9AF9CFB0]
R14: [0x00007F0CF85F2420]
R15: [0x00007F0CF806F260]
EFL: [0x0000000000000206]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

Regards,

Zijian

[gcusello]

Hi @zijian,

every time I have a crash on one Splunk system I open a case to Splunk Support sending them a diag of the server.

Especially when you have so frequent crashes on both two production indexers because the service continuity is in danger: I'd open a case with priority 2 or 1.

Ciao.

Giuseppe