Splunk crashing

[steveo2]

Hi,

We recently upgraded from 7.3.1 to 8.1.4.  Shortly after we have seen Splunkd crashing.  The crash log is below.  Any help would be greatly appreciated. Thanks 

[build 17f862b42a7c] 2021-06-14 15:24:21
Access violation, cannot read at address [0x00000000000003C0]
Exception address: [0x00007FF6A9059B66]
Crashing thread: indexerPipe
MxCsr: [0x0000000000001FA0]
SegDs: [0x000000000000002B]
SegEs: [0x000000000000002B]
SegFs: [0x0000000000000053]
SegGs: [0x000000000000002B]
SegSs: [0x000000000000002B]
SegCs: [0x0000000000000033]
EFlags: [0x0000000000010202]
Rsp: [0x000000610CAFEED0]
Rip: [0x00007FF6A9059B66] ?
Dr0: [0x0000000000000000]
Dr1: [0x0000000000000000]
Dr2: [0x0000000000000000]
Dr3: [0x0000000000000000]
Dr6: [0x0000000000000000]
Dr7: [0x0000000000000000]
Rax: [0x00000000000003A8]
Rcx: [0x0000000000000000]
Rdx: [0x0000000000000003]
Rbx: [0x00000000000003A8]
Rbp: [0x000000610CAFF030]
Rsi: [0x0000000000000003]
Rdi: [0x000000610CAFF4C8]
R8: [0x000000610CAFF4C8]
R9: [0x0000000000000000]
R10: [0x000000610CAFF000]
R11: [0x00000210282052E0]
R12: [0x0000000000000024]
R13: [0x0000021028205320]
R14: [0x000000610CAFF024]
R15: [0x0000000000000000]
DebugControl: [0x0000021000000001]
LastBranchToRip: [0x0000000000000000]
LastBranchFromRip: [0x0000000000000000]
LastExceptionToRip: [0x0000000000000000]
LastExceptionFromRip: [0x0000000000000000]

OS: Windows
Arch: x86-64

Backtrace:
[0x00007FF6A9059B66] ?
Args: [0x0000021028205344] [0x000000610CAFF4C8] [0x00000210256EA5D0]
[0x00007FF6A9033055] ?
Args: [0x0000021028BAB7E0] [0x0000000000000000] [0x0000000060C31E4D]
[0x00007FF6A9032956] ?
Args: [0x000000610CAFF730] [0x000000610CAFFA38] [0x0000000060C31E4D]
[0x00007FF6A9048795] ?
Args: [0x01D75E9B8846C480] [0x00000210266D6ED0] [0x00000210209CF980]
[0x00007FF6A907F4C1] ?
Args: [0x0000000000000000] [0x000000610CAFF780] [0x00007FF6AB85C701]
[0x00007FF6A907EDC5] ?
Args: [0x000000610CAFFA38] [0x000000610CAFFA38] [0x00000210209CF980]
[0x00007FF6A907ECC5] ?
Args: [0x0000000060C774A5] [0x0000000000000001] [0x0000000060C774A5]
[0x00007FF6A908A218] ?
Args: [0x00000210243F2558] [0x000000610CAFFAC0] [0x00000210209CF980]
[0x00007FF6A961CF2E] ?
Args: [0x00000210243F2558] [0x0000000000000001] [0x0000000000000000]
[0x00007FF6A973964D] ?
Args: [0x00000210205AF7D0] [0x00000210205AF7D0] [0x0000000000000000]
[0x00007FF6A9969F5E] ?
Args: [0x0000000000000000] [0x0000000000000000] [0x0000000000000000]
[0x00007FFA0651FB80] o__realloc_base + 96/528
Args: [0x0000000000000000] [0x0000000000000000] [0x0000000000000000]
[0x00007FFA069A84D4] BaseThreadInitThunk + 20/1472
Args: [0x0000000000000000] [0x0000000000000000] [0x0000000000000000]
[0x00007FFA090E1791] RtlUserThreadStart + 33/784
Args: [0x0000000000000000] [0x0000000000000000] [0x0000000000000000]
Crash dump written to: D:\Splunk\var\log\splunk\D__Splunk_bin_splunkd_exe_crash-2021-06-14-15-24-21.dmp

Splunk ran as local administrator
SIPR-SPLUNK2 /Windows Server 10.0 or later
GetLastError(): 5
Threads running: 90
Executable module base: 0x0x7ff6a8c40000
Runtime: 41.899370s
argv: [Splunkd -p 8089]
Thread: "indexerPipe", did_join=0, ready_to_run=Y, main_thread=N, token=4180

x86 CPUID registers:
0: 00000015 756E6547 6C65746E 49656E69
1: 00050654 00010800 FEDA3203 0F8BFBFF
2: 76036301 00F0B6FF 00000000 00C30000
3: 00000000 00000000 00000000 00000000
4: 00000121 01C0003F 0000003F 00000000
5: 00000000 00000000 00000000 00000000
6: 00000000 00000000 00000000 00000000
7: 00000000 D09F6FB9 00000000 9C000400
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 00000000 00000000 00000000 00000000
B: 00000000 00000001 00000100 00000000
C: 00000000 00000000 00000000 00000000
😧 000000FF 00000A80 00000A80 00000000
E: 00000000 00000000 00000000 00000000
F: 00000000 00000000 00000000 00000000
10: 00000000 00000000 00000000 00000000
11: 00000000 00000000 00000000 00000000
12: 00000000 00000000 00000000 00000000
13: 00000000 00000000 00000000 00000000
14: 00000000 00000000 00000000 00000000
15: 00000000 00000000 00000000 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000121 2C100800
80000002: 65746E49 2952286C 6F655820 2952286E
80000003: 6C695320 20726576 30313134 55504320
80000004: 32204020 4730312E 00007A48 00000000
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000000
80000008: 0000302C 00000000 00000000 00000000
terminating...

[richgalloway · Accepted solution]

Open a support case with Splunk.