Monitoring Splunk

Splunkd crash after upgrade to 5.0.1

rush05
Engager

Hi, we just upgraded splunk from 4.3.1 to 5.0.1 and now every time we start splunk, splunkd crashes.

Here is the text from the crashlog:

[build 143156] 2013-01-21 11:29:10
Received fatal signal 6 (Abort).
 Cause:
   Unknown signal origin (si_code=-1).
 Crashing thread: MainTailingThread
 Registers:
    RIP:  [0xFFFFFD7FFEAE314A] __lwp_kill + 10 (/lib/amd64/libc.so.1)
    RDI:  [0x0000000000000032]
    RSI:  [0x0000000000000006]
    RBP:  [0xFFFFFD7FF53FF350]
    RSP:  [0xFFFFFD7FF53FF348]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x0000000000000005]
    RDX:  [0xFFFFFEBFB492CBE0]
    R8:  [0x0000000000000001]
    R9:  [0x0000000000000000]
    R10:  [0x0000000000000005]
    R11:  [0x0000000000000020]
    R12:  [0x00000000000000E5]
    R13:  [0x0000000001CC4E98]
    R14:  [0x0000000001AEA230]
    R15:  [0xFFFFFD7FF53FF840]
    RFL:  [0x0000000000000286]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000004]
    CS:  [0x000000000000004B]
    GS:  [0x0000000000000000]
    FS:  [0x0000000000000000]

 OS: SunOS
 Arch: x86-64

 Backtrace:
  [0xFFFFFFFFFFFFFFFF] ?
  [0xFFFFFD7FFEA87F19] raise + 25 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEA669CE] abort + 94 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEA66C8E] _assert + 126 (/lib/amd64/libc.so.1)
  [0x00000000009D1148] _ZN16FileInputTracker10computeCRCEPm14FileDescriptorRK3Strll + 920 (/opt/splunk/bin/splunkd)
  [0x00000000009D1656] _ZN16FileInputTracker11fileHalfMd5EPm14FileDescriptorRK3Strll + 22 (/opt/splunk/bin/splunkd)
  [0x00000000009EA99A] _ZN3WTF13loadFishStateEb + 650 (/opt/splunk/bin/splunkd)
  [0x00000000009DDAB4] _ZN10TailReader8readFileER15WatchedTailFileP11TailWatcher + 148 (/opt/splunk/bin/splunkd)
  [0x00000000009DDCDE] _ZN11TailWatcher8readFileER15WatchedTailFile + 254 (/opt/splunk/bin/splunkd)
  [0x00000000009DFD92] _ZN11TailWatcher11fileChangedEP16WatchedFileStateRK7Timeval + 482 (/opt/splunk/bin/splunkd)
  [0x0000000000EBFC84] _ZN30FilesystemChangeInternalWorker15callFileChangedER7TimevalP16WatchedFileState + 148 (/opt/splunk/bin/splunkd)
  [0x0000000000EC1743] _ZN30FilesystemChangeInternalWorker12when_expiredERy + 499 (/opt/splunk/bin/splunkd)
  [0x0000000000F18634] _ZN11TimeoutHeap18runExpiredTimeoutsER7Timeval + 180 (/opt/splunk/bin/splunkd)
  [0x0000000000EBABBC] _ZN9EventLoop3runEv + 188 (/opt/splunk/bin/splunkd)
  [0x00000000009E6B70] _ZN11TailWatcher3runEv + 144 (/opt/splunk/bin/splunkd)
  [0x00000000009E6CD6] _ZN13TailingThread4mainEv + 278 (/opt/splunk/bin/splunkd)
  [0x0000000000F16012] _ZN6Thread8callMainEPv + 98 (/opt/splunk/bin/splunkd)
  [0xFFFFFD7FFEADD60B] _thr_slot_offset + 795 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEADD840] smt_pause + 96 (/lib/amd64/libc.so.1)
 SunOS / dtslabmgt04 / 5.10 / Generic_147441-24 / i86pc
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2013-01-18 10:10:25.499 -0500 Interrupt signal received
    2013-01-18 11:53:06.183 -0500 splunkd started (build 115073)
    2013-01-21 10:23:51.077 -0500 Interrupt signal received
    2013-01-21 11:26:26.127 -0500 splunkd started (build 143156)
    Assertion failed: bytesToHash < 1048576, file /opt/splunk/p4/splunk/branches/5.0.1/src/pipeline/input/FileInputTracker.cpp, line 229
    2013-01-21 11:27:50.445 -0500 splunkd started (build 143156)
    Assertion failed: bytesToHash < 1048576, file /opt/splunk/p4/splunk/branches/5.0.1/src/pipeline/input/FileInputTracker.cpp, line 229
    2013-01-21 11:29:08.706 -0500 splunkd started (build 143156)
    Assertion failed: bytesToHash < 1048576, file /opt/splunk/p4/splunk/branches/5.0.1/src/pipeline/input/FileInputTracker.cpp, line 229

Threads running: 40
argv: [splunkd -p 8089 start]
terminating...

If anyone can help, it would be much appreciated. This is on our test server, but we are supposed to roll it out into production soon.

Thanks!

1 Solution

jbsplunk
Splunk Employee
Splunk Employee

This is a known issue, SPL-58292, which has been reported to support and is presently being investigated. At this time, there aren't any workarounds.

View solution in original post

jbsplunk
Splunk Employee
Splunk Employee

This is a known issue, SPL-58292, which has been reported to support and is presently being investigated. At this time, there aren't any workarounds.

jbsplunk
Splunk Employee
Splunk Employee

You can file a case with support if you have an enterprise contract and they'll be able to provide you with additional details.

0 Karma

rush05
Engager

Thank you for your reply. Where would I go to find more details on SPL-58292?

0 Karma
Get Updates on the Splunk Community!

Splunk Observability for AI

Don’t miss out on an exciting Tech Talk on Splunk Observability for AI!Discover how Splunk’s agentic AI ...

Splunk Enterprise Security 8.x: The Essential Upgrade for Threat Detection, ...

Watch On Demand the Tech Talk on November 6 at 11AM PT, and empower your SOC to reach new heights! Duration: ...

Splunk Observability as Code: From Zero to Dashboard

For the details on what Self-Service Observability and Observability as Code is, we have some awesome content ...