Forwarder keeps crashing

[djcmay]

We have on Server with 3 forwarders installed. One of those are not working anymore. It keeps crashing shortly after restarting:

[build 182037] 2014-03-19 09:44:40
Received fatal signal 6 (Abort).
 Cause:
   Unknown signal origin (si_code=-1).
 Crashing thread: archivereader
 Registers:
    PC:  [0xFFFFFFFF796DCB68] __lwp_kill + 8 (/lib/sparcv9/libc.so.1)
    nPC:  [0xFFFFFFFF796DCB6C] __lwp_kill + 12 (/lib/sparcv9/libc.so.1)
    Y:  [0x0000000000000000]
    G1:  [0x00000000000000A3]
    G2:  [0x0000000101208A4C]
    G3:  [0x0000000000000000]
    G4:  [0x0000000000000000]
    G5:  [0xFFFFFFFFFF7FFFFF]
    G6:  [0x0000000000000000]
    G7:  [0xFFFFFFFF7A60D200]
    O0:  [0x0000000000000000]
    O1:  [0x0000000000000006]
    O2:  [0xFFFFFFFF7BE08334]
    O3:  [0xFFFFFFFF7964C178]
    O4:  [0x0000000000000005]
    O5:  [0xFFFFFFFF7984EC60]
    O6:  [0xFFFFFFFF740F2471]
    O7:  [0xFFFFFFFF7967443C]

 OS: SunOS
 Arch: SPARC

 Backtrace:
  [0xFFFFFFFF7967443C] raise + 16 (/lib/sparcv9/libc.so.1)
  [0xFFFFFFFF7964C178] abort + 208 (/lib/sparcv9/libc.so.1)
  [0xFFFFFFFF7964C464] _assert + 116 (/lib/sparcv9/libc.so.1)
  [0x0000000100382DA4] _ZN17ArchiveCrcChecker21seekAndComputeSeekCrcEv + 740 (/pkg/PRSL/fPRSLFB/splunkforwarder/bin/splunkd)
  [0x0000000100386E1C] _ZN17ArchiveCrcChecker5writeEPKcm + 492 (/pkg/PRSL/fPRSLFB/splunkforwarder/bin/splunkd)
  [0x0000000101101274] _ZN17ArchiveCrcChecker5writeEPKvm + 12 (/pkg/PRSL/fPRSLFB/splunkforwarder/bin/splunkd)
  [0x0000000100634758] _ZN14ArchiveContext7processERK8PathnameP13ISourceWriter + 688 (/pkg/PRSL/fPRSLFB/splunkforwarder/bin/splunkd)
  [0x0000000100635188] _ZN14ArchiveContext9readFullyEP13ISourceWriterRb + 1080 (/pkg/PRSL/fPRSLFB/splunkforwarder/bin/splunkd)
  [0x0000000100385840] _ZN16ArchiveProcessor20haveReadAsNonArchiveE14FileDescriptorlPK3Str + 512 (/pkg/PRSL/fPRSLFB/splunkforwarder/bin/splunkd)
  [0x0000000100388E78] _ZN16ArchiveProcessor4mainEv + 4048 (/pkg/PRSL/fPRSLFB/splunkforwarder/bin/splunkd)
  [0x00000001009BA560] _ZN6Thread8callMainEPv + 128 (/pkg/PRSL/fPRSLFB/splunkforwarder/bin/splunkd)
  [0xFFFFFFFF796D8AFC] _thr_slot_offset + 1160 (/lib/sparcv9/libc.so.1)
 SunOS / tpm2s120 / 5.10 / Generic_148888-01 / sun4u
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    Assertion failed: (file_offset_t)_seekPtr >= dp->curPos(), file /opt/splunk/p4/splunk/branches/6.0.0/src/pipeline/input/ArchiveProcessor.cpp, line 1044
    2014-03-17 15:38:48.849 +0100 splunkd started (build 182037)
    Assertion failed: (file_offset_t)_seekPtr >= dp->curPos(), file /opt/splunk/p4/splunk/branches/6.0.0/src/pipeline/input/ArchiveProcessor.cpp, line 1044
    2014-03-17 15:46:00.719 +0100 splunkd started (build 182037)
    Assertion failed: (file_offset_t)_seekPtr >= dp->curPos(), file /opt/splunk/p4/splunk/branches/6.0.0/src/pipeline/input/ArchiveProcessor.cpp, line 1044
    2014-03-19 09:44:11.873 +0100 splunkd started (build 182037)
    Assertion failed: (file_offset_t)_seekPtr >= dp->curPos(), file /opt/splunk/p4/splunk/branches/6.0.0/src/pipeline/input/ArchiveProcessor.cpp, line 1044

Last errno: 0
Threads running: 29
argv: [splunkd -p 8090 start]
Thread: "archivereader", did_join=0, ready_to_run=Y, main_thread=N
First 4 bytes of Thread token @104399bfc:
00000000  00 00 00 1c                                       |....|
00000004
terminating...

Does anybody got an idea, what the problem is?

[MuS · Accepted solution]

Hi djcmay,

This is a known issue and fixed as of 6.0.1

Upgrading causes crash in "Crashing Thread: archivereader" (SPL-74873)

cheers, MuS