Hi
Recently i upgraded my splunk single instance from 7.2.2 to 8.1.0. splunkd keep on crashing everyday at specific time around 6PM. When i checked the latest crash log i got the below error.
Received fatal signal 11 (Segmentation fault). Cause: Signal sent by kernel. Crashing thread: archivereader Registers: RIP: [0x0000564C2B743B40] _ZN14CharacterClass9set_multiEPKcmb + 32 (splunkd + 0x21D4B40)
.
OS: Linux Arch: x86-64
Backtrace (PIC build): [0x0000564C2B743B40] _ZN14CharacterClass9set_multiEPKcmb + 32 (splunkd + 0x21D4B40) [0x0000564C2B2DFA45] _ZN27STDataInputHeaderProcessing21performPostProcessingEP11PipelineSetR12PipelineData + 69 (splunkd + 0x1D70A45) [0x0000564C2AB9BFEF] _ZN16ArchiveProcessor29performSTDataHeaderProcessingEv + 47 (splunkd + 0x162CFEF) [0x0000564C2AB9C23C] _ZN16ArchiveProcessor10writeEventEPKcm + 492 (splunkd + 0x162D23C) [0x0000564C2AB9E6CF] _ZN16ArchiveProcessor22awaitingClassificationEPKcm + 287 (splunkd + 0x162F6CF) [0x0000564C2AB9E741] _ZN16ArchiveProcessor5writeEPKvm + 65 (splunkd + 0x162F741) [0x0000564C2B14453C] _ZN14ArchiveContext7processERK8PathnameP13ISourceWriter + 940 (splunkd + 0x1BD553C) [0x0000564C2B144CA0] _ZN14ArchiveContext9readFullyEP13ISourceWriterRb + 1200 (splunkd + 0x1BD5CA0) [0x0000564C2ABA1141] _ZN16ArchiveProcessor14processArchiveER5CRC_tS1_ + 5489 (splunkd + 0x1632141) [0x0000564C2AA2ECC6] _ZN16ArchiveProcessor4mainEv + 614 (splunkd + 0x14BFCC6) [0x0000564C2B830627] _ZN6Thread8callMainEPv + 135 (splunkd + 0x22C1627) [0x00007F4C3AFB0EA5] ? (libpthread.so.0 + 0x7EA5) [0x00007F4C3ACD9B0D] clone + 109 (libc.so.6 + 0xFEB0D) Linux / security01.dca.int.untd.com / 4.20.5-1.el7.elrepo.x86_64 / #1 SMP Sat Jan 26 10:55:51 EST 2019 / x86_64 /etc/redhat-release: CentOS Linux release 7.9.2009 (Core) glibc version: 2.17 glibc release: stable
.
Last errno: 0 Threads running: 82 Runtime: 19747.282336s argv: [splunkd -p 8089 restart splunkd] Regex JIT enabled
RE2 regex engine enabled
using CLOCK_MONOTONIC Thread: "archivereader", did_join=0, ready_to_run=Y, main_thread=N, token=139964947887872 MutexByte: MutexByte-waiting={none}
x86 CPUID registers: 0: 0000000D 756E6547 6C65746E 49656E69 1: 000306F2 02400800 FEFA3203 1FCBFBFF
.
80000008: 0000302E 00000000 00000000 00000000 terminating... Here is the var/log/message: Jul 4 06:00:22 hostname kernel: [10196539.758876] traps: splunkd[7701] general protection fault ip:563a1f215b40 sp:7f21cc3f6070 error:0 in splunkd[563a1d041000+408d000]
Can someone please provide a solution for this.
... View more
