Splunk universal forwarder crashing - Crashing thread: parsing

[Splunker8]

Splunk universal forwarder crashes

here are crash logs:

[build de405f4a7979] 2023-07-10 17:31:30
Received fatal signal 11 (Segmentation fault) on PID 3013854.
Cause:
No memory mapped at address [0x0000000000000080].
Crashing thread: parsing
Registers:
RIP: [0x00007FBC41EDEA74] __pthread_mutex_lock + 4 (libpthread.so.0 + 0xAA74)
RDI: [0x0000000000000070]
RSI: [0x00007FBC3E21A0B0]
RBP: [0x00007FBC2FDFD980]
RSP: [0x00007FBC2FDFD8C8]
RAX: [0x0000558B2F9877E0]
RBX: [0x0000000000000000]
RCX: [0x0000000000000000]
RDX: [0x00007FBC2FDFD8F8]
R8: [0x0000000000000000]
R9: [0x00007FBC41200080]
R10: [0x00000000000000A3]
R11: [0x0000000000000000]
R12: [0x0000000000000001]
R13: [0x0000000000000070]
R14: [0x00007FBC2FDFD8F0]
R15: [0x0000558B2F9877D0]
EFL: [0x0000000000010202]
TRAPNO: [0x000000000000000E]
ERR: [0x0000000000000004]
CSGSFS: [0x002B000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x00007FBC41EDEA74] __pthread_mutex_lock + 4 (libpthread.so.0 + 0xAA74)
[0x0000558B2CE030D9] _ZN16PthreadMutexImpl4lockEv + 9 (splunkd + 0x2DD20D9)
[0x0000558B2CD3ED27] _ZN9EventLoop20internal_runInThreadEP13InThreadActorb + 103 (splunkd + 0x2D0DD27)
[0x0000558B2CB7B19A] _ZN11Distributed11EloopRunner3runEPNS_15EloopRunnerTaskE + 170 (splunkd + 0x2B4A19A)
[0x0000558B2C02A6A6] _ZN18TcpOutputProcessor7executeER15CowPipelineData + 230 (splunkd + 0x1FF96A6)
[0x0000558B2C7B1B29] _ZN9Processor12executeMultiER18PipelineDataVectorPS0_ + 73 (splunkd + 0x2780B29)
[0x0000558B2BDA03A2] _ZN8Pipeline4mainEv + 1074 (splunkd + 0x1D6F3A2)
[0x0000558B2CE02DAD] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 13 (splunkd + 0x2DD1DAD)
[0x0000558B2CE03CA2] _ZN6Thread8callMainEPv + 178 (splunkd + 0x2DD2CA2)
[0x00007FBC41EDC1CF] ? (libpthread.so.0 + 0x81CF)
[0x00007FBC4146ADD3] clone + 67 (libc.so.6 + 0x39DD3)

[inventsekar]

Splunk UF crashing logs analysis is a difficult task.. only Splunk Support guys can do that. if you have the support contract, pls make a support ticket..

also just for learning purposes...

1) the UF version, linux or win, pls

2) did you do any UF upgrade recently? 

3) any new config files, did you push from DS to this UF?

4) do you face this UF crash issue only one UF or multiple UFs

[Splunker8]

1.
UF 9.0.4  
OS: Linux
Arch: x86-64

2. 
No upgrade done

3.
No changes. Its a new installation.

4.
crashing on only one UF

[inventsekar]

4.
crashing on only one UF 

.... if you are looking for short answer... uninstall that 9.0.4 UF, use another UF version... either 9.0.0 or 9.1.0..etc..

if you are looking for a perfect solution, then, Splunk support ticket is the only answer. (9.0.4 UF is a recent one... the linux and that UF may have some compatibility issues or.. that particular linux is giving some troubles to the UF.. only Splunk Support guys can solve this issues.. thanks. 

[richgalloway]

Submit a support request.