Why does my Splunk universal forwarder crashe : Crashing thread: parsing

[ips_mandar]

I have splunk universal forwarder version-6.5.2 after few days it crashes and gives error as-

Received fatal signal 6 (Aborted).
 Cause:
   Signal sent by PID 27417 running under UID 558.
 Crashing thread: parsing
 Registers:
    RIP:  [0x00007F05A1DAE5F7] gsignal + 55 (libc.so.6 + 0x355F7)
    RDI:  [0x0000000000006B19]
    RSI:  [0x0000000000006B71]
    RBP:  [0x00007F05A2134868]
    RSP:  [0x00007F05961FE628]
    RAX:  [0x0000000000000000]
    RBX:  [0x00007F05A4F01197]
    RCX:  [0xFFFFFFFFFFFFFFFF]
    RDX:  [0x0000000000000006]
    R8:  [0x000000000000000A]
    R9:  [0x00007F05961FF700]
    R10:  [0x0000000000000008]
    R11:  [0x0000000000000202]
    R12:  [0x0000000000000000]
    R13:  [0x0000000000000000]
    R14:  [0x00007F05961FEDB0]
    R15:  [0x0000000000000000]
    EFL:  [0x0000000000000202]
    TRAPNO:  [0x0000000000000000]
    ERR:  [0x0000000000000000]
    CSGSFS:  [0x0000000000000033]
    OLDMASK:  [0x0000000000000000]
 OS: Linux
 Arch: x86-64
 Backtrace (PIC build):
 Linux / inc3037.nxdi.nl-cdc01.nxp.com / 3.10.0-514.6.2.el7.x86_64 / #1 SMP Fri Feb 17 19:21:31 EST 2017 / x86_64
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    WARN - The file '/var/lib/splunkforwarder/var/log/splunk/splunkd_stderr.log' is invalid. Reason: std::bad_alloc
    ERROR - Error (std::bad_alloc) encountered while getting file type for '/var/log/syslog/2019/03/29/abv04.log'.
    ERROR - Error (std::bad_alloc) encountered while getting file type for '/var/log/syslog/2019/03/29/apy76.log'.
    ERROR - Error (std::bad_alloc) encountered while getting file type for '/var/log/syslog/2019/03/29/inf7.log'.
    ERROR - Error (std::bad_alloc) encountered while getting file type for '/var/lib/splunkforwarder/var/log/splunk/splunkd_stderr.log'.
    WARN - The file '/var/lib/splunkforwarder/var/log/splunk/splunkd_stderr.log' is invalid. Reason: std::bad_alloc
    terminate called after throwing an instance of 'St9bad_alloc'
      what():  std::bad_alloc
 /etc/redhat-release: Red Hat Enterprise Linux Server release 7.2 (Maipo)
 glibc version: 2.17
 glibc release: stable
Last errno: 12
Threads running: 35
Runtime: 22180.910445s
argv: [splunkd -p 8089 restart]
Regex JIT enabled
Thread: "parsing", did_join=0, ready_to_run=Y, main_thread=N
First 8 bytes of Thread token @0x7f0596210010:
00000000  00 f7 1f 96 05 7f 00 00                           |........|
00000008
x86 CPUID registers:
         0: 0000000F 756E6547 6C65746E 49656E69
         1: 000306F2 04100800 7FFEFBFF BFEBFBFF
         2: 76036301 00F0B5FF 00000000 00C10000
         3: 00000000 00000000 00000000 00000000
         4: 00000000 00000000 00000000 00000000
         5: 00000040 00000040 00000003 00002120
         6: 00000077 00000002 00000009 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000001 00000000 00000000 00000000
         A: 07300403 00000000 00000000 00000603
         B: 00000000 00000000 000000CD 00000004
         C: 00000000 00000000 00000000 00000000
         😧 00000000 00000000 00000000 00000000
         E: 00000000 00000000 00000000 00000000
         F: 00000000 00000000 00000000 00000000
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000021 2C100800
  80000002: 65746E49 2952286C 6F655820 2952286E
  80000003: 55504320 2D354520 33343632 20337620
  80000004: 2E332040 48473034 0000007A 00000000
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 0000302E 00000000 00000000 00000000
terminating...

Gives error like std::bad_alloc ...any help will be appreciated.
Thanks,

[ddrillic]

The following speaks about the std::bad_alloc error - Error:std::bad_alloc Whenever I try to visualize a search

[ips_mandar]

Thanks @ddrillic, I do have memory available but still it is crashing after two-three days by giving error as std::bad_alloc and it is happening in my universal forwarder which is sending syslog data .

[ips_mandar]

Today it shows crashing thread as Crashing thread: TcpOutEloop in splunkd_crash_log

[fatmaBouaziz]

Hello i Have the same problem. my Heavy forwarder is restarting unexpectedly at least 2 times a day.

crash.log file specifies the thread : 

"Received fatal signal 6 (Aborted) on PID 13698.
Cause:
Signal sent by PID 13698 running under UID 0.
Crashing thread: TcpOutEloop"

Did you manage to solve the problem please?

[alec_stan]

Hello

Did you manage to resolve, I am facing a similar problem with my intermediate forwarders crushing.

I have enough memory 16GB

[PickleRick]

Are you still using a 6.5 forwarder in 2024? I suppose not. Try starting a new thread with a more detailed description of your problem.

[alec_stan]

No, I am using 9.1 and 9.3. However, I am experiencing the same problem on Red Hat 8. I will start a new thread as per your advice.