topic Re: Splunk universal forwarder crashing - Crashing thread: parsing in Getting Data In

Splunk universal forwarder crashing - Crashing thread: parsing

Splunker8 — Tue, 11 Jul 2023 22:38:05 GMT

Splunk universal forwarder crashes

here are crash logs:

[build de405f4a7979] 2023-07-10 17:31:30
Received fatal signal 11 (Segmentation fault) on PID 3013854.
Cause:
No memory mapped at address [0x0000000000000080].
Crashing thread: parsing
Registers:
RIP: [0x00007FBC41EDEA74] __pthread_mutex_lock + 4 (libpthread.so.0 + 0xAA74)
RDI: [0x0000000000000070]
RSI: [0x00007FBC3E21A0B0]
RBP: [0x00007FBC2FDFD980]
RSP: [0x00007FBC2FDFD8C8]
RAX: [0x0000558B2F9877E0]
RBX: [0x0000000000000000]
RCX: [0x0000000000000000]
RDX: [0x00007FBC2FDFD8F8]
R8: [0x0000000000000000]
R9: [0x00007FBC41200080]
R10: [0x00000000000000A3]
R11: [0x0000000000000000]
R12: [0x0000000000000001]
R13: [0x0000000000000070]
R14: [0x00007FBC2FDFD8F0]
R15: [0x0000558B2F9877D0]
EFL: [0x0000000000010202]
TRAPNO: [0x000000000000000E]
ERR: [0x0000000000000004]
CSGSFS: [0x002B000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x00007FBC41EDEA74] __pthread_mutex_lock + 4 (libpthread.so.0 + 0xAA74)
[0x0000558B2CE030D9] _ZN16PthreadMutexImpl4lockEv + 9 (splunkd + 0x2DD20D9)
[0x0000558B2CD3ED27] _ZN9EventLoop20internal_runInThreadEP13InThreadActorb + 103 (splunkd + 0x2D0DD27)
[0x0000558B2CB7B19A] _ZN11Distributed11EloopRunner3runEPNS_15EloopRunnerTaskE + 170 (splunkd + 0x2B4A19A)
[0x0000558B2C02A6A6] _ZN18TcpOutputProcessor7executeER15CowPipelineData + 230 (splunkd + 0x1FF96A6)
[0x0000558B2C7B1B29] _ZN9Processor12executeMultiER18PipelineDataVectorPS0_ + 73 (splunkd + 0x2780B29)
[0x0000558B2BDA03A2] _ZN8Pipeline4mainEv + 1074 (splunkd + 0x1D6F3A2)
[0x0000558B2CE02DAD] _ZN6Thread37_callMainAndDiscardTerminateExceptionEv + 13 (splunkd + 0x2DD1DAD)
[0x0000558B2CE03CA2] _ZN6Thread8callMainEPv + 178 (splunkd + 0x2DD2CA2)
[0x00007FBC41EDC1CF] ? (libpthread.so.0 + 0x81CF)
[0x00007FBC4146ADD3] clone + 67 (libc.so.6 + 0x39DD3)

Re: Splunk universal forwarder crashing - Crashing thread: parsing

richgalloway — Wed, 12 Jul 2023 00:39:33 GMT

Submit a support request.

Re: Splunk universal forwarder crashing - Crashing thread: parsing

inventsekar — Wed, 12 Jul 2023 01:27:10 GMT

Splunk UF crashing logs analysis is a difficult task.. only Splunk Support guys can do that. if you have the support contract, pls make a support ticket..

also just for learning purposes...

1) the UF version, linux or win, pls

2) did you do any UF upgrade recently?

3) any new config files, did you push from DS to this UF?

4) do you face this UF crash issue only one UF or multiple UFs

Re: Splunk universal forwarder crashing - Crashing thread: parsing

Splunker8 — Wed, 12 Jul 2023 11:39:24 GMT

1.
UF 9.0.4
OS: Linux
Arch: x86-64

2.
No upgrade done

3.
No changes. Its a new installation.

4.
crashing on only one UF

Re: Splunk universal forwarder crashing - Crashing thread: parsing

inventsekar — Thu, 13 Jul 2023 00:16:11 GMT

4.
crashing on only one UF

.... if you are looking for short answer... uninstall that 9.0.4 UF, use another UF version... either 9.0.0 or 9.1.0..etc..

if you are looking for a perfect solution, then, Splunk support ticket is the only answer. (9.0.4 UF is a recent one... the linux and that UF may have some compatibility issues or.. that particular linux is giving some troubles to the UF.. only Splunk Support guys can solve this issues.. thanks.