Why does Splunkd keeps crashing? Received fatal signal 11 (Segmentation fault)

[krishna6677]

Hi

Recently i upgraded my splunk single instance from 7.2.2 to 8.1.0.

splunkd keep on crashing everyday at specific time around 6PM. When i checked the latest crash log i got the below error.

Received fatal signal 11 (Segmentation fault).
Cause:
Signal sent by kernel.
Crashing thread: archivereader
Registers:
RIP: [0x0000564C2B743B40] _ZN14CharacterClass9set_multiEPKcmb + 32 (splunkd + 0x21D4B40)

.

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x0000564C2B743B40] _ZN14CharacterClass9set_multiEPKcmb + 32 (splunkd + 0x21D4B40)
[0x0000564C2B2DFA45] _ZN27STDataInputHeaderProcessing21performPostProcessingEP11PipelineSetR12PipelineData + 69 (splunkd + 0x1D70A45)
[0x0000564C2AB9BFEF] _ZN16ArchiveProcessor29performSTDataHeaderProcessingEv + 47 (splunkd + 0x162CFEF)
[0x0000564C2AB9C23C] _ZN16ArchiveProcessor10writeEventEPKcm + 492 (splunkd + 0x162D23C)
[0x0000564C2AB9E6CF] _ZN16ArchiveProcessor22awaitingClassificationEPKcm + 287 (splunkd + 0x162F6CF)
[0x0000564C2AB9E741] _ZN16ArchiveProcessor5writeEPKvm + 65 (splunkd + 0x162F741)
[0x0000564C2B14453C] _ZN14ArchiveContext7processERK8PathnameP13ISourceWriter + 940 (splunkd + 0x1BD553C)
[0x0000564C2B144CA0] _ZN14ArchiveContext9readFullyEP13ISourceWriterRb + 1200 (splunkd + 0x1BD5CA0)
[0x0000564C2ABA1141] _ZN16ArchiveProcessor14processArchiveER5CRC_tS1_ + 5489 (splunkd + 0x1632141)
[0x0000564C2AA2ECC6] _ZN16ArchiveProcessor4mainEv + 614 (splunkd + 0x14BFCC6)
[0x0000564C2B830627] _ZN6Thread8callMainEPv + 135 (splunkd + 0x22C1627)
[0x00007F4C3AFB0EA5] ? (libpthread.so.0 + 0x7EA5)
[0x00007F4C3ACD9B0D] clone + 109 (libc.so.6 + 0xFEB0D)
Linux / security01.dca.int.untd.com / 4.20.5-1.el7.elrepo.x86_64 / #1 SMP Sat Jan 26 10:55:51 EST 2019 / x86_64
/etc/redhat-release: CentOS Linux release 7.9.2009 (Core)
glibc version: 2.17
glibc release: stable

.

Last errno: 0
Threads running: 82
Runtime: 19747.282336s
argv: [splunkd -p 8089 restart splunkd]
Regex JIT enabled

RE2 regex engine enabled

using CLOCK_MONOTONIC
Thread: "archivereader", did_join=0, ready_to_run=Y, main_thread=N, token=139964947887872
MutexByte: MutexByte-waiting={none}

x86 CPUID registers:
0: 0000000D 756E6547 6C65746E 49656E69
1: 000306F2 02400800 FEFA3203 1FCBFBFF

.

80000008: 0000302E 00000000 00000000 00000000
terminating...

Here is the var/log/message: Jul 4 06:00:22 hostname kernel: [10196539.758876] traps: splunkd[7701] general protection fault ip:563a1f215b40 sp:7f21cc3f6070 error:0 in splunkd[563a1d041000+408d000]

Can someone please provide a solution for this.