Regular Segmentation fault on indexer

[au_chrismor]

Over the last week or so one of my dev indexers keeps crashing with a signal 11:

How can I start to figure out what's happening?  I know I didn't change anything on the system.  I've attached the log from the most recent dump.  Page locations vary but the basic fault remains the same:

[build 0b8d769cb912] 2025-02-21 12:43:55
Received fatal signal 11 (Segmentation fault) on PID 552462.
Cause:
No memory mapped at address [0x00007742D7E337C0].
Crashing thread: IndexerTPoolWorker-1
Registers:
RIP: [0x00005DA750CCC63E] _ZNK23PipelineDataMetaKeyAtom16asIndexableTokenEP15st_token_answerRK10StrSegmentPK20PipelineInputChannelRK19PipelineDataMetaKeyb + 126 (splunkd + 0x2A7863E)
RDI: [0x00007746F1FFE2B8]
RSI: [0x00007742D7E337D0]
RBP: [0x00007746F1FFE330]
RSP: [0x00007746F1FFE290]
RAX: [0x00007746F1FFE2B8]
RBX: [0x00007746D80B7B08]
RCX: [0x000000000000000B]
RDX: [0x000000000000000B]
R8: [0x00007746D80B7B30]
R9: [0x0000000000000001]
R10: [0x00007746F1FFDA20]
R11: [0x0000000000000004]
R12: [0x00007746F1FFE410]
R13: [0x00007746F1FFE410]
R14: [0x00007746B5A6D968]
R15: [0x00007746D80B7B30]
EFL: [0x0000000000010246]
TRAPNO: [0x000000000000000E]
ERR: [0x0000000000000004]
CSGSFS: [0x002B000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x00005DA750CCC63E] _ZNK23PipelineDataMetaKeyAtom16asIndexableTokenEP15st_token_answerRK10StrSegmentPK20PipelineInputChannelRK19PipelineDataMetaKeyb + 126 (splunkd + 0x2A7863E)
[0x00005DA7506F0599] _ZN10PutterBase7putMetaERK15CowPipelineDataNS_23indexed_fields_policy_tEb + 249 (splunkd + 0x249C599)
[0x00005DA7506F0BDC] _ZN14IndexableValue15indexIntoPutterER10PutterBase + 76 (splunkd + 0x249CBDC)
[0x00005DA7506F0E0F] _ZN14IndexableValue5indexEPN5STMgr14HandleWritableEPN9Segmenter7ContextE + 191 (splunkd + 0x249CE0F)
[0x00005DA7507A1208] _ZN11StreamGroup3runEm + 296 (splunkd + 0x254D208)
[0x00005DA7501FBF68] _ZN6Worker4mainEv + 184 (splunkd + 0x1FA7F68)
[0x00005DA75343C3AE] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 46 (splunkd + 0x51E83AE)
[0x00005DA75343C4BB] _ZN6Thread8callMainEPv + 139 (splunkd + 0x51E84BB)
[0x000077470029CAA4] ? (libc.so.6 + 0x74AA4)
[0x0000774700329C3C] ? (libc.so.6 + 0x101C3C)
Linux / splunk / 6.8.0-53-generic / #55-Ubuntu SMP PREEMPT_DYNAMIC Fri Jan 17 15:37:52 UTC 2025 / x86_64
/etc/debian_version: trixie/sid
MAP: 5da74e254000-5da75487b000 r-xp 00000000 fc:00 793619 /opt/splunk/bin/splunkd
MAP: 5da75487b000-5da7549b6000 r--p 06626000 fc:00 793619 /opt/splunk/bin/splunkd
MAP: 5da7549b6000-5da7549db000 rw-p 06761000 fc:00 793619 /opt/splunk/bin/splunkd
MAP: 5da7549db000-5da754af1000 rw-p 00000000 00:00 0
MAP: 77469a800000-77469b000000 rw-p 00000000 00:00 0
MAP: 77469b000000-77469b001000 ---p 00000000 00:00 0
MAP: 77469b001000-77469b201000 rw-p 00000000 00:00 0
MAP: 77469b400000-7746a7800000 rw-p 00000000 00:00 0
MAP: 7746a7800000-7746a7801000 ---p 00000000 00:00 0
MAP: 7746a7801000-7746a7a01000 rw-p 00000000 00:00 0
MAP: 7746a7c00000-7746b3000000 rw-p 00000000 00:00 0
MAP: 7746b3000000-7746b3001000 ---p 00000000 00:00 0
MAP: 7746b3001000-7746b3201000 rw-p 00000000 00:00 0
MAP: 7746b3400000-7746d6e00000 rw-p 00000000 00:00 0
MAP: 7746d6e00000-7746d6e01000 ---p 00000000 00:00 0
MAP: 7746d6e01000-7746d7001000 rw-p 00000000 00:00 0
MAP: 7746d7200000-7746db400000 rw-p 00000000 00:00 0
MAP: 7746db400000-7746db401000 ---p 00000000 00:00 0
MAP: 7746db401000-7746db601000 rw-p 00000000 00:00 0
MAP: 7746db800000-7746db801000 ---p 00000000 00:00 0
MAP: 7746db801000-7746dba01000 rw-p 00000000 00:00 0
MAP: 7746dbc00000-7746dec00000 rw-p 00000000 00:00 0
MAP: 7746dec00000-7746dec01000 ---p 00000000 00:00 0
MAP: 7746dec01000-7746dee01000 rw-p 00000000 00:00 0
MAP: 7746df000000-7746df200000 rw-p 00000000 00:00 0
MAP: 7746df200000-7746df201000 ---p 00000000 00:00 0
MAP: 7746df201000-7746df401000 rw-p 00000000 00:00 0
MAP: 7746df600000-7746dfa00000 rw-p 00000000 00:00 0
MAP: 7746dfa00000-7746dfa01000 ---p 00000000 00:00 0
MAP: 7746dfa01000-7746dfc01000 rw-p 00000000 00:00 0
MAP: 7746dfe00000-7746dfe01000 ---p 00000000 00:00 0
MAP: 7746dfe01000-7746e0001000 rw-p 00000000 00:00 0
MAP: 7746e0200000-7746e0201000 ---p 00000000 00:00 0
MAP: 7746e0201000-7746e0401000 rw-p 00000000 00:00 0
MAP: 7746e0600000-7746e0601000 ---p 00000000 00:00 0
MAP: 7746e0601000-7746e0801000 rw-p 00000000 00:00 0
MAP: 7746e0a00000-7746e0a01000 ---p 00000000 00:00 0
MAP: 7746e0a01000-7746e0c01000 rw-p 00000000 00:00 0
MAP: 7746e0e00000-7746e0e01000 ---p 00000000 00:00 0
MAP: 7746e0e01000-7746e1001000 rw-p 00000000 00:00 0
MAP: 7746e1200000-7746e1400000 rw-p 00000000 00:00 0
MAP: 7746e1400000-7746e1401000 ---p 00000000 00:00 0
MAP: 7746e1401000-7746e1601000 rw-p 00000000 00:00 0
MAP: 7746e1800000-7746e1c00000 rw-p 00000000 00:00 0
MAP: 7746e1c00000-7746e1c01000 ---p 00000000 00:00 0
MAP: 7746e1c01000-7746e1e01000 rw-p 00000000 00:00 0
MAP: 7746e2000000-7746e2001000 ---p 00000000 00:00 0
MAP: 7746e2001000-7746e2201000 rw-p 00000000 00:00 0
MAP: 7746e2400000-7746e2401000 ---p 00000000 00:00 0
MAP: 7746e2401000-7746e2601000 rw-p 00000000 00:00 0
MAP: 7746e2800000-7746e3200000 rw-p 00000000 00:00 0
MAP: 7746e3200000-7746e3201000 ---p 00000000 00:00 0
MAP: 7746e3201000-7746e3401000 rw-p 00000000 00:00 0
MAP: 7746e3600000-7746e3601000 ---p 00000000 00:00 0
MAP: 7746e3601000-7746e3801000 rw-p 00000000 00:00 0
MAP: 7746e3a00000-7746e3a01000 ---p 00000000 00:00 0
MAP: 7746e3a01000-7746e3c01000 rw-p 00000000 00:00 0
MAP: 7746e3e00000-7746e3e01000 ---p 00000000 00:00 0
MAP: 7746e3e01000-7746e4001000 rw-p 00000000 00:00 0
MAP: 7746e4200000-7746e4201000 ---p 00000000 00:00 0
MAP: 7746e4201000-7746e4401000 rw-p 00000000 00:00 0
MAP: 7746e4600000-7746e4601000 ---p 00000000 00:00 0
MAP: 7746e4601000-7746e4801000 rw-p 00000000 00:00 0
MAP: 7746e4a00000-7746e4a01000 ---p 00000000 00:00 0
MAP: 7746e4a01000-7746e4c01000 rw-p 00000000 00:00 0
MAP: 7746e4e00000-7746e4e01000 ---p 00000000 00:00 0
MAP: 7746e4e01000-7746e5001000 rw-p 00000000 00:00 0
MAP: 7746e5200000-7746e5201000 ---p 00000000 00:00 0
MAP: 7746e5201000-7746e5401000 rw-p 00000000 00:00 0
MAP: 7746e5600000-7746e5601000 ---p 00000000 00:00 0
MAP: 7746e5601000-7746e5801000 rw-p 00000000 00:00 0
MAP: 7746e5a00000-7746e5a01000 ---p 00000000 00:00 0
MAP: 7746e5a01000-7746e5c01000 rw-p 00000000 00:00 0
MAP: 7746e5e00000-7746e5e01000 ---p 00000000 00:00 0
MAP: 7746e5e01000-7746e6001000 rw-p 00000000 00:00 0
MAP: 7746e6200000-7746e6201000 ---p 00000000 00:00 0
MAP: 7746e6201000-7746e6401000 rw-p 00000000 00:00 0
MAP: 7746e6600000-7746e6601000 ---p 00000000 00:00 0
MAP: 7746e6601000-7746e6801000 rw-p 00000000 00:00 0
MAP: 7746e6a00000-7746e6a01000 ---p 00000000 00:00 0
MAP: 7746e6a01000-7746e6c01000 rw-p 00000000 00:00 0
MAP: 7746e6e00000-7746e6e01000 ---p 00000000 00:00 0
MAP: 7746e6e01000-7746e7001000 rw-p 00000000 00:00 0
MAP: 7746e7200000-7746e7201000 ---p 00000000 00:00 0
MAP: 7746e7201000-7746e7401000 rw-p 00000000 00:00 0
MAP: 7746e7600000-7746e7800000 rw-p 00000000 00:00 0
MAP: 7746e7800000-7746e7801000 ---p 00000000 00:00 0
MAP: 7746e7801000-7746e7a01000 rw-p 00000000 00:00 0
MAP: 7746e7c00000-7746e7e00000 rw-p 00000000 00:00 0
MAP: 7746e7e00000-7746e7e01000 ---p 00000000 00:00 0
MAP: 7746e7e01000-7746e8001000 rw-p 00000000 00:00 0
MAP: 7746e8200000-7746e8201000 ---p 00000000 00:00 0
MAP: 7746e8201000-7746e8401000 rw-p 00000000 00:00 0
MAP: 7746e8600000-7746e8c00000 rw-p 00000000 00:00 0
MAP: 7746e8c00000-7746e8c01000 ---p 00000000 00:00 0
MAP: 7746e8c01000-7746e8e01000 rw-p 00000000 00:00 0
MAP: 7746e9000000-7746e9001000 ---p 00000000 00:00 0
MAP: 7746e9001000-7746e9201000 rw-p 00000000 00:00 0
MAP: 7746e9400000-7746e9401000 ---p 00000000 00:00 0
MAP: 7746e9401000-7746e9601000 rw-p 00000000 00:00 0
MAP: 7746e9800000-7746e9801000 ---p 00000000 00:00 0
MAP: 7746e9801000-7746e9a01000 rw-p 00000000 00:00 0
MAP: 7746e9c00000-7746e9c01000 ---p 00000000 00:00 0
MAP: 7746e9c01000-7746ea401000 rw-p 00000000 00:00 0
MAP: 7746ea600000-7746ea601000 ---p 00000000 00:00 0
MAP: 7746ea601000-7746eae01000 rw-p 00000000 00:00 0
MAP: 7746eb000000-7746eb001000 ---p 00000000 00:00 0
MAP: 7746eb001000-7746eb801000 rw-p 00000000 00:00 0
MAP: 7746eba00000-7746eba01000 ---p 00000000 00:00 0
MAP: 7746eba01000-7746ec201000 rw-p 00000000 00:00 0
MAP: 7746ec400000-7746ec401000 ---p 00000000 00:00 0
MAP: 7746ec401000-7746ec601000 rw-p 00000000 00:00 0
MAP: 7746ec800000-7746ec801000 ---p 00000000 00:00 0
MAP: 7746ec801000-7746eca01000 rw-p 00000000 00:00 0
MAP: 7746ecc00000-7746ecc01000 ---p 00000000 00:00 0
MAP: 7746ecc01000-7746ece01000 rw-p 00000000 00:00 0
MAP: 7746ed000000-7746ed001000 ---p 00000000 00:00 0
MAP: 7746ed001000-7746ed201000 rw-p 00000000 00:00 0
MAP: 7746ed400000-7746ed401000 ---p 00000000 00:00 0
MAP: 7746ed401000-7746ed601000 rw-p 00000000 00:00 0
MAP: 7746ed800000-7746ed801000 ---p 00000000 00:00 0
MAP: 7746ed801000-7746eda01000 rw-p 00000000 00:00 0
MAP: 7746edc00000-7746edc01000 ---p 00000000 00:00 0
MAP: 7746edc01000-7746ede01000 rw-p 00000000 00:00 0
MAP: 7746ee000000-7746ee001000 ---p 00000000 00:00 0
MAP: 7746ee001000-7746ee201000 rw-p 00000000 00:00 0
MAP: 7746ee400000-7746ee401000 ---p 00000000 00:00 0
MAP: 7746ee401000-7746ee601000 rw-p 00000000 00:00 0
MAP: 7746ee800000-7746ee801000 ---p 00000000 00:00 0
MAP: 7746ee801000-7746eea01000 rw-p 00000000 00:00 0
MAP: 7746eec00000-7746eec01000 ---p 00000000 00:00 0
MAP: 7746eec01000-7746eee01000 rw-p 00000000 00:00 0
MAP: 7746ef000000-7746f0e00000 rw-p 00000000 00:00 0
MAP: 7746f0e00000-7746f0e01000 ---p 00000000 00:00 0
MAP: 7746f0e01000-7746f1001000 rw-p 00000000 00:00 0
MAP: 7746f1200000-7746f1201000 ---p 00000000 00:00 0
MAP: 7746f1201000-7746f1401000 rw-p 00000000 00:00 0
MAP: 7746f1600000-7746f1601000 ---p 00000000 00:00 0
MAP: 7746f1601000-7746f1801000 rw-p 00000000 00:00 0
MAP: 7746f1a00000-7746f1a01000 ---p 00000000 00:00 0
MAP: 7746f1a01000-7746f1c01000 rw-p 00000000 00:00 0
MAP: 7746f1e00000-7746f1e01000 ---p 00000000 00:00 0
MAP: 7746f1e01000-7746f2001000 rw-p 00000000 00:00 0
MAP: 7746f2200000-7746f2201000 ---p 00000000 00:00 0
MAP: 7746f2201000-7746f2401000 rw-p 00000000 00:00 0
MAP: 7746f2600000-7746f2601000 ---p 00000000 00:00 0
MAP: 7746f2601000-7746f2801000 rw-p 00000000 00:00 0
MAP: 7746f2a00000-7746f2a01000 ---p 00000000 00:00 0
MAP: 7746f2a01000-7746f2c01000 rw-p 00000000 00:00 0
MAP: 7746f2e00000-7746f2e01000 ---p 00000000 00:00 0
MAP: 7746f2e01000-7746f3001000 rw-p 00000000 00:00 0
MAP: 7746f3200000-7746f3400000 rw-p 00000000 00:00 0
MAP: 7746f3400000-7746f3401000 ---p 00000000 00:00 0
MAP: 7746f3401000-7746f3601000 rw-p 00000000 00:00 0
MAP: 7746f3800000-7746f3c00000 rw-p 00000000 00:00 0
MAP: 7746f3c00000-7746f3c01000 ---p 00000000 00:00 0
MAP: 7746f3c01000-7746f3e01000 rw-p 00000000 00:00 0
MAP: 7746f4000000-7746f4001000 ---p 00000000 00:00 0
MAP: 7746f4001000-7746f4201000 rw-p 00000000 00:00 0
MAP: 7746f4400000-7746f4401000 ---p 00000000 00:00 0
MAP: 7746f4401000-7746f4601000 rw-p 00000000 00:00 0
MAP: 7746f4800000-7746f4801000 ---p 00000000 00:00 0
MAP: 7746f4801000-7746f4a01000 rw-p 00000000 00:00 0
MAP: 7746f4c00000-7746f4e00000 rw-p 00000000 00:00 0
MAP: 7746f4e00000-7746f4e01000 ---p 00000000 00:00 0
MAP: 7746f4e01000-7746f5001000 rw-p 00000000 00:00 0
MAP: 7746f5200000-7746f5400000 rw-p 00000000 00:00 0
MAP: 7746f5400000-7746f5401000 ---p 00000000 00:00 0
MAP: 7746f5401000-7746f5601000 rw-p 00000000 00:00 0
MAP: 7746f5800000-7746f6000000 rw-p 00000000 00:00 0
MAP: 7746f6000000-7746f6001000 ---p 00000000 00:00 0
MAP: 7746f6001000-7746f6201000 rw-p 00000000 00:00 0
MAP: 7746f6400000-7746f6600000 rw-p 00000000 00:00 0
MAP: 7746f6600000-7746f6601000 ---p 00000000 00:00 0
MAP: 7746f6601000-7746f6801000 rw-p 00000000 00:00 0
MAP: 7746f6a00000-7746f6c00000 rw-p 00000000 00:00 0
MAP: 7746f6c00000-7746f6c01000 ---p 00000000 00:00 0
MAP: 7746f6c01000-7746f6e01000 rw-p 00000000 00:00 0
MAP: 7746f7000000-7746f7400000 rw-p 00000000 00:00 0
MAP: 7746f7400000-7746f7401000 ---p 00000000 00:00 0
MAP: 7746f7401000-7746f7601000 rw-p 00000000 00:00 0
MAP: 7746f7800000-7746f7801000 ---p 00000000 00:00 0
MAP: 7746f7801000-7746f7a01000 rw-p 00000000 00:00 0
MAP: 7746f7c00000-7746f7e00000 rw-p 00000000 00:00 0
MAP: 7746f7e00000-7746f7e01000 ---p 00000000 00:00 0
MAP: 7746f7e01000-7746f8001000 rw-p 00000000 00:00 0
MAP: 7746f8200000-7746f8400000 rw-p 00000000 00:00 0
MAP: 7746f8400000-7746f8401000 ---p 00000000 00:00 0
MAP: 7746f8401000-7746f8601000 rw-p 00000000 00:00 0
MAP: 7746f8800000-7746f8a00000 rw-p 00000000 00:00 0
MAP: 7746f8a00000-7746f8a01000 ---p 00000000 00:00 0
MAP: 7746f8a01000-7746f8c01000 rw-p 00000000 00:00 0
MAP: 7746f8e00000-7746f8e01000 ---p 00000000 00:00 0
MAP: 7746f8e01000-7746f9001000 rw-p 00000000 00:00 0
MAP: 7746f9200000-7746f9400000 rw-p 00000000 00:00 0
MAP: 7746f9400000-7746f9401000 ---p 00000000 00:00 0
MAP: 7746f9401000-7746f9601000 rw-p 00000000 00:00 0
MAP: 7746f9800000-7746fb000000 rw-p 00000000 00:00 0
MAP: 7746fb000000-7746fb001000 ---p 00000000 00:00 0
MAP: 7746fb001000-7746fb201000 rw-p 00000000 00:00 0
MAP: 7746fb400000-7746fb800000 rw-p 00000000 00:00 0
MAP: 7746fb800000-7746fb801000 ---p 00000000 00:00 0
MAP: 7746fb801000-7746fba01000 rw-p 00000000 00:00 0
MAP: 7746fbc00000-7746fc000000 rw-p 00000000 00:00 0
MAP: 7746fc000000-7746fc001000 ---p 00000000 00:00 0
MAP: 7746fc001000-7746fc201000 rw-p 00000000 00:00 0
MAP: 7746fc400000-7746fec00000 rw-p 00000000 00:00 0
MAP: 7746fec00000-7746fec01000 ---p 00000000 00:00 0
MAP: 7746fec01000-7746fee01000 rw-p 00000000 00:00 0
MAP: 7746fee70000-7746ff000000 rwxp 00000000 00:00 0
MAP: 7746ff000000-7746ff200000 rw-p 00000000 00:00 0
MAP: 7746ff200000-7746ff201000 ---p 00000000 00:00 0
MAP: 7746ff201000-7746ff401000 rw-p 00000000 00:00 0
MAP: 7746ff410000-7746ff600000 rwxp 00000000 00:00 0
MAP: 7746ff600000-7746ffa00000 rw-p 00000000 00:00 0
MAP: 7746ffa09000-7746ffa0c000 r--p 00000000 fc:00 3027055 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
MAP: 7746ffa0c000-7746ffa13000 r-xp 00003000 fc:00 3027055 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
MAP: 7746ffa13000-7746ffa15000 r--p 0000a000 fc:00 3027055 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
MAP: 7746ffa15000-7746ffa16000 r--p 0000b000 fc:00 3027055 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
MAP: 7746ffa16000-7746ffa17000 rw-p 0000c000 fc:00 3027055 /usr/lib/x86_64-linux-gnu/libnuma.so.1.0.0
MAP: 7746ffa17000-7746ffb47000 rwxp 00000000 00:00 0
MAP: 7746ffb47000-7746ffb4b000 r--p 00000000 fc:00 3018177 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
MAP: 7746ffb4b000-7746ffb6f000 r-xp 00004000 fc:00 3018177 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
MAP: 7746ffb6f000-7746ffb73000 r--p 00028000 fc:00 3018177 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
MAP: 7746ffb73000-7746ffb74000 r--p 0002b000 fc:00 3018177 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
MAP: 7746ffb74000-7746ffb75000 rw-p 0002c000 fc:00 3018177 /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
MAP: 7746ffb75000-7746ffb76000 ---p 00000000 00:00 0
MAP: 7746ffb76000-7746ffb7f000 rw-p 00000000 00:00 0
MAP: 7746ffb7f000-7746ffb80000 ---p 00000000 00:00 0
MAP: 7746ffb80000-7746ffe00000 rwxp 00000000 00:00 0
MAP: 7746ffe00000-774700200000 rw-p 00000000 00:00 0
MAP: 774700200000-774700228000 r--p 00000000 fc:00 3024032 /usr/lib/x86_64-linux-gnu/libc.so.6
MAP: 774700228000-7747003b0000 r-xp 00028000 fc:00 3024032 /usr/lib/x86_64-linux-gnu/libc.so.6
MAP: 7747003b0000-7747003ff000 r--p 001b0000 fc:00 3024032 /usr/lib/x86_64-linux-gnu/libc.so.6
MAP: 7747003ff000-774700403000 r--p 001fe000 fc:00 3024032 /usr/lib/x86_64-linux-gnu/libc.so.6
MAP: 774700403000-774700405000 rw-p 00202000 fc:00 3024032 /usr/lib/x86_64-linux-gnu/libc.so.6
MAP: 774700405000-774700412000 rw-p 00000000 00:00 0
MAP: 774700413000-7747004e3000 rwxp 00000000 00:00 0
MAP: 7747004e3000-774700518000 rw-p 00000000 00:00 0
MAP: 774700518000-774700528000 r--p 00000000 fc:00 3024035 /usr/lib/x86_64-linux-gnu/libm.so.6
MAP: 774700528000-7747005a7000 r-xp 00010000 fc:00 3024035 /usr/lib/x86_64-linux-gnu/libm.so.6
MAP: 7747005a7000-7747005ff000 r--p 0008f000 fc:00 3024035 /usr/lib/x86_64-linux-gnu/libm.so.6
MAP: 7747005ff000-774700600000 r--p 000e7000 fc:00 3024035 /usr/lib/x86_64-linux-gnu/libm.so.6
MAP: 774700600000-774700601000 rw-p 000e8000 fc:00 3024035 /usr/lib/x86_64-linux-gnu/libm.so.6
MAP: 774700601000-774700603000 rw-p 00000000 00:00 0
MAP: 774700603000-77470060e000 r--p 00000000 fc:00 801871 /opt/splunk/lib/libxslt.so.1.1.34
MAP: 77470060e000-774700648000 r-xp 0000b000 fc:00 801871 /opt/splunk/lib/libxslt.so.1.1.34
MAP: 774700648000-774700655000 r--p 00045000 fc:00 801871 /opt/splunk/lib/libxslt.so.1.1.34
MAP: 774700655000-774700656000 r--p 00051000 fc:00 801871 /opt/splunk/lib/libxslt.so.1.1.34
MAP: 774700656000-774700657000 rw-p 00052000 fc:00 801871 /opt/splunk/lib/libxslt.so.1.1.34
MAP: 774700657000-774700658000 rw-p 00000000 00:00 0
MAP: 774700658000-774700687000 r--p 00000000 fc:00 801162 /opt/splunk/lib/libxml2.so.2.9.10
MAP: 774700687000-7747007c8000 r-xp 0002f000 fc:00 801162 /opt/splunk/lib/libxml2.so.2.9.10
MAP: 7747007c8000-77470080b000 r--p 00170000 fc:00 801162 /opt/splunk/lib/libxml2.so.2.9.10
MAP: 77470080b000-774700813000 r--p 001b2000 fc:00 801162 /opt/splunk/lib/libxml2.so.2.9.10
MAP: 774700813000-774700815000 rw-p 001ba000 fc:00 801162 /opt/splunk/lib/libxml2.so.2.9.10
MAP: 774700815000-774700816000 rw-p 00000000 00:00 0
MAP: 774700816000-7747008c1000 r-xp 00000000 fc:00 801863 /opt/splunk/lib/libarchive.so.13.6.2
MAP: 7747008c1000-7747008c2000 ---p 000ab000 fc:00 801863 /opt/splunk/lib/libarchive.so.13.6.2
MAP: 7747008c2000-7747008c5000 r--p 000ab000 fc:00 801863 /opt/splunk/lib/libarchive.so.13.6.2
MAP: 7747008c5000-7747008c6000 rw-p 000ae000 fc:00 801863 /opt/splunk/lib/libarchive.so.13.6.2
MAP: 7747008c6000-7747008c7000 rw-p 00000000 00:00 0
MAP: 7747008c7000-774700988000 r-xp 00000000 fc:00 809367 /opt/splunk/lib/libpcre2-8.so.0.11.0
MAP: 774700988000-774700989000 ---p 000c1000 fc:00 809367 /opt/splunk/lib/libpcre2-8.so.0.11.0
MAP: 774700989000-77470098a000 r--p 000c1000 fc:00 809367 /opt/splunk/lib/libpcre2-8.so.0.11.0
MAP: 77470098a000-77470098b000 rw-p 000c2000 fc:00 809367 /opt/splunk/lib/libpcre2-8.so.0.11.0
MAP: 77470098b000-7747009a5000 r--p 00000000 fc:00 801865 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7747009a5000-7747009e4000 r-xp 0001a000 fc:00 801865 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7747009e4000-7747009f6000 r--p 00059000 fc:00 801865 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7747009f6000-7747009fa000 r--p 0006a000 fc:00 801865 /opt/splunk/lib/libssl.so.1.0.0
MAP: 7747009fa000-774700a00000 rw-p 0006e000 fc:00 801865 /opt/splunk/lib/libssl.so.1.0.0
MAP: 774700a00000-774700a82000 r--p 00000000 fc:00 801160 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 774700a82000-774700c10000 r-xp 00082000 fc:00 801160 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 774700c10000-774700caf000 r--p 00210000 fc:00 801160 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 774700caf000-774700ccd000 r--p 002af000 fc:00 801160 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 774700ccd000-774700cdc000 rw-p 002cd000 fc:00 801160 /opt/splunk/lib/libcrypto.so.1.0.0
MAP: 774700cdc000-774700ce0000 rw-p 00000000 00:00 0
MAP: 774700ce0000-774700ce2000 r--p 00000000 fc:00 793681 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 774700ce2000-774700cef000 r-xp 00002000 fc:00 793681 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 774700cef000-774700cf1000 r--p 0000f000 fc:00 793681 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 774700cf1000-774700cf2000 r--p 00010000 fc:00 793681 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 774700cf2000-774700cf3000 rw-p 00011000 fc:00 793681 /opt/splunk/lib/libbz2.so.1.0.3
MAP: 774700cf3000-774700d0d000 r-xp 00000000 fc:00 809363 /opt/splunk/lib/libz.so.1.2.11
MAP: 774700d0d000-774700d0e000 r--p 00019000 fc:00 809363 /opt/splunk/lib/libz.so.1.2.11
MAP: 774700d0e000-774700d0f000 rw-p 0001a000 fc:00 809363 /opt/splunk/lib/libz.so.1.2.11
MAP: 774700d0f000-774700d11000 rw-p 00000000 00:00 0
MAP: 774700d11000-774700d12000 r--p 00000000 fc:00 3024046 /usr/lib/x86_64-linux-gnu/librt.so.1
MAP: 774700d12000-774700d13000 r-xp 00001000 fc:00 3024046 /usr/lib/x86_64-linux-gnu/librt.so.1
MAP: 774700d13000-774700d14000 r--p 00002000 fc:00 3024046 /usr/lib/x86_64-linux-gnu/librt.so.1
MAP: 774700d14000-774700d15000 r--p 00002000 fc:00 3024046 /usr/lib/x86_64-linux-gnu/librt.so.1
MAP: 774700d15000-774700d16000 rw-p 00003000 fc:00 3024046 /usr/lib/x86_64-linux-gnu/librt.so.1
MAP: 774700d16000-774700d17000 r--p 00000000 fc:00 3024034 /usr/lib/x86_64-linux-gnu/libdl.so.2
MAP: 774700d17000-774700d18000 r-xp 00001000 fc:00 3024034 /usr/lib/x86_64-linux-gnu/libdl.so.2
MAP: 774700d18000-774700d19000 r--p 00002000 fc:00 3024034 /usr/lib/x86_64-linux-gnu/libdl.so.2
MAP: 774700d19000-774700d1a000 r--p 00002000 fc:00 3024034 /usr/lib/x86_64-linux-gnu/libdl.so.2
MAP: 774700d1a000-774700d1b000 rw-p 00003000 fc:00 3024034 /usr/lib/x86_64-linux-gnu/libdl.so.2
MAP: 774700d1b000-774700d1c000 r--p 00000000 fc:00 3024044 /usr/lib/x86_64-linux-gnu/libpthread.so.0
MAP: 774700d1c000-774700d1d000 r-xp 00001000 fc:00 3024044 /usr/lib/x86_64-linux-gnu/libpthread.so.0
MAP: 774700d1d000-774700d1e000 r--p 00002000 fc:00 3024044 /usr/lib/x86_64-linux-gnu/libpthread.so.0
MAP: 774700d1e000-774700d1f000 r--p 00002000 fc:00 3024044 /usr/lib/x86_64-linux-gnu/libpthread.so.0
MAP: 774700d1f000-774700d20000 rw-p 00003000 fc:00 3024044 /usr/lib/x86_64-linux-gnu/libpthread.so.0
MAP: 774700d25000-774700d26000 rw-p 00000000 00:00 0
MAP: 774700d26000-774700ea9000 r-xp 00000000 fc:00 809362 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 774700ea9000-774700eac000 r--p 00182000 fc:00 809362 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 774700eac000-774700eb3000 rw-p 00185000 fc:00 809362 /opt/splunk/lib/libsqlite3.so.0.8.6
MAP: 774700eb3000-774700eb5000 rw-p 00000000 00:00 0
MAP: 774700eb5000-774700ec3000 r--p 00000000 fc:00 793683 /opt/splunk/lib/libxmlsec1-openssl.so.1.2.24
MAP: 774700ec3000-774700efd000 r-xp 0000e000 fc:00 793683 /opt/splunk/lib/libxmlsec1-openssl.so.1.2.24
MAP: 774700efd000-774700f08000 r--p 00048000 fc:00 793683 /opt/splunk/lib/libxmlsec1-openssl.so.1.2.24
MAP: 774700f08000-774700f0b000 r--p 00052000 fc:00 793683 /opt/splunk/lib/libxmlsec1-openssl.so.1.2.24
MAP: 774700f0b000-774700f0c000 rw-p 00055000 fc:00 793683 /opt/splunk/lib/libxmlsec1-openssl.so.1.2.24
MAP: 774700f0c000-774700f0d000 rw-p 00000000 00:00 0
MAP: 774700f0d000-774700f1f000 r--p 00000000 fc:00 809364 /opt/splunk/lib/libxmlsec1.so.1.2.24
MAP: 774700f1f000-774700f71000 r-xp 00012000 fc:00 809364 /opt/splunk/lib/libxmlsec1.so.1.2.24
MAP: 774700f71000-774700f86000 r--p 00064000 fc:00 809364 /opt/splunk/lib/libxmlsec1.so.1.2.24
MAP: 774700f86000-774700f88000 r--p 00079000 fc:00 809364 /opt/splunk/lib/libxmlsec1.so.1.2.24
MAP: 774700f88000-774700f8a000 rw-p 0007b000 fc:00 809364 /opt/splunk/lib/libxmlsec1.so.1.2.24
MAP: 774700f8a000-774700fc2000 r-xp 00000000 fc:00 801862 /opt/splunk/lib/libbson-1.0.so.0.0.0
MAP: 774700fc2000-774700fc3000 ---p 00038000 fc:00 801862 /opt/splunk/lib/libbson-1.0.so.0.0.0
MAP: 774700fc3000-774700fc6000 r--p 00038000 fc:00 801862 /opt/splunk/lib/libbson-1.0.so.0.0.0
MAP: 774700fc6000-774700fc7000 rw-p 0003b000 fc:00 801862 /opt/splunk/lib/libbson-1.0.so.0.0.0
MAP: 774700fc7000-774700fcd000 rw-p 00000000 00:00 0
MAP: 774700fcd000-774701083000 r-xp 00000000 fc:00 809361 /opt/splunk/lib/libmongoc-1.0.so.0.0.0
MAP: 774701083000-774701084000 r--p 000b5000 fc:00 809361 /opt/splunk/lib/libmongoc-1.0.so.0.0.0
MAP: 774701084000-774701087000 rw-p 000b6000 fc:00 809361 /opt/splunk/lib/libmongoc-1.0.so.0.0.0
MAP: 774701087000-7747010ce000 r-xp 00000000 fc:00 801857 /opt/splunk/lib/libjemalloc.so.2
MAP: 7747010ce000-7747010cf000 ---p 00047000 fc:00 801857 /opt/splunk/lib/libjemalloc.so.2
MAP: 7747010cf000-7747010d1000 r--p 00047000 fc:00 801857 /opt/splunk/lib/libjemalloc.so.2
MAP: 7747010d1000-7747010d2000 rw-p 00049000 fc:00 801857 /opt/splunk/lib/libjemalloc.so.2
MAP: 7747010d2000-7747010d5000 rw-p 00000000 00:00 0
MAP: 7747010d5000-7747010d6000 r--p 00000000 fc:00 3024029 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
MAP: 7747010d6000-774701101000 r-xp 00001000 fc:00 3024029 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
MAP: 774701101000-77470110b000 r--p 0002c000 fc:00 3024029 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
MAP: 77470110b000-77470110d000 r--p 00036000 fc:00 3024029 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
MAP: 77470110d000-77470110f000 rw-p 00038000 fc:00 3024029 /usr/lib/x86_64-linux-gnu/ld-linux-x86-64.so.2
MAP: 7fffa4a7d000-7fffa4a9e000 rw-p 00000000 00:00 0 [stack]
MAP: 7fffa4b9b000-7fffa4b9f000 r--p 00000000 00:00 0 [vvar]
MAP: 7fffa4b9f000-7fffa4ba1000 r-xp 00000000 00:00 0 [vdso]
MAP: ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]
Last errno: 2
Threads running: 85
Runtime: 19.367557s
argv: [splunkd -p 8089 restart]
Regex JIT enabled

RE2 regex engine enabled

using CLOCK_MONOTONIC
Thread: "IndexerTPoolWorker-1", did_join=0, ready_to_run=Y, main_thread=N, token=131146591504064
MutexByte: MutexByte-waiting={none}
TPool Worker: _isExecutorWorker=N, _id=1
Running TJob: name=TJob


x86 CPUID registers:
0: 00000016 756E6547 6C65746E 49656E69
1: 000A0653 01000800 FFFAB223 0F8BFBFF
2: 00000001 00000000 0000004D 002C307D
3: 00000000 00000000 00000000 00000000
4: 00000121 01C0003F 0000003F 00000001
5: 00000000 00000000 00000003 00000000
6: 00000004 00000000 00000000 00000000
7: 00000000 009C47AB 00000004 BC000400
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300402 00000000 00000000 00008603
B: 00000000 00000001 00000100 00000001
C: 00000000 00000000 00000000 00000000
😧 0000001F 00000440 00000440 00000000
E: 00000000 00000000 00000000 00000000
F: 00000000 00000000 00000000 00000000
10: 00000000 00000000 00000000 00000000
11: 00000000 00000000 00000000 00000000
12: 00000000 00000000 00000000 00000000
13: 00000000 00000000 00000000 00000000
14: 00000000 00000000 00000000 00000000
15: 00000000 00000000 00000000 00000000
16: 00000000 00000000 00000000 00000000
80000000: 80000008 756E6547 6C65746E 49656E69
80000001: 000A0653 00000000 00000121 2C100800
80000002: 65746E49 2952286C 726F4320 4D542865
80000003: 35692029 3630312D 43203030 40205550
80000004: 332E3320 7A484730 00000000 00000000
80000005: 01FF01FF 01FF01FF 40020140 40020140
80000006: 00000000 42004200 02008140 00808140
80000007: 00000000 00000000 00000000 00000000
80000008: 00003027 0100D000 00000000 00000000
terminating...

[au_chrismor]

I'm going to work my way through all the suggestions.  Since both of the replies suggested THP settings, I'll start there.

Thanks

[richgalloway]

Check the ulimits settings to make they meet or exceed Splunk's recommendations.

Verify THP is disabled.

If both of the above pass then open a case with Splunk Support.

[livehybrid]

A segmentation fault (signal 11) in Splunk can have several potential causes, including memory corruption, insufficient resources, software bugs, or issues with the system configuration. Since you mentioned that you haven't changed anything recently, it's crucial to systematically investigate and rule out potential causes.

Steps to Troubleshoot and Diagnose the Issue:

1. Check Splunk Logs for More Clues

Look at the splunkd.log and crash.log files for additional context around the crash. These logs can be found in:

$SPLUNK_HOME/var/log/splunk/splunkd.log
$SPLUNK_HOME/var/log/splunk/crash*.log

Run:

grep -i 'fatal' $SPLUNK_HOME/var/log/splunk/splunkd.log 
grep -i 'segfault' $SPLUNK_HOME/var/log/splunk/crash*.log

This might provide more context on what was happening before the crash.

 

2. Validate System Memory and Kernel Overcommit Settings

Your crash log shows "No memory mapped at address", which suggests possible memory issues. Check for kernel memory overcommitting, which can lead to random segmentation faults.

Run:

cat /proc/sys/vm/overcommit_memory

• If the value is 0, memory overcommit handling is heuristic-based.
• If 1, the system allows overcommitting memory, which is not recommended for Splunk.
• If 2, it's strict (recommended).

If it's set to 1, you could consider changing it to see if that effects the Splunk service:

echo 2 | sudo tee /proc/sys/vm/overcommit_memory

And persist the change in /etc/sysctl.conf:

vm.overcommit_memory = 2

Also see https://splunk.my.site.com/customer/s/article/Indexer-crashed-after-OS-upgrade

 

3. Check Transparent Huge Pages (THP)

THP can cause issues with Splunk's memory management. Disable it if it’s enabled.

Check the current status:

cat /sys/kernel/mm/transparent_hugepage/enabled

If it says [always], disable it temporarily:

echo never | sudo tee /sys/kernel/mm/transparent_hugepage/enabled 
echo never | sudo tee /sys/kernel/mm/transparent_hugepage/defrag

To make it permanent, add the following to /etc/rc.local:

echo never > /sys/kernel/mm/transparent_hugepage/enabled 
echo never > /sys/kernel/mm/transparent_hugepage/defrag

4. Check ulimits for Splunk User

If the indexer is running into resource exhaustion, check its ulimits:

ulimit -a

Try updating to the following if not already:

nofile = 65536
nproc = 16384

Adjust in /etc/security/limits.conf:

splunk soft nofile 65536
splunk hard nofile 65536 
splunk soft nproc 16384 
splunk hard nproc 16384

5. Check Splunk’s Memory and CPU Usage

Run:

ps aux --sort=-%mem | grep splunk 
free -m 
top -o %CPU

Look for excessive memory or CPU consumption.

 

6. Check for Recent Software Updates or Kernel Changes

If the system has undergone automatic updates, it might have introduced compatibility issues.

Check recent updates:

cat /var/log/dpkg.log | grep -i "upgrade" # Debian/Ubuntu 
cat /var/log/yum.log | grep -i "update" # RHEL/CentOS

Please let me know how you get on and consider accepting this answer or adding karma this answer if it has helped.
Regards

Will