Splunk Search

Splunk Crashing once in 10min


Hi All,

My Splunk instance 5.0.1 running in Solaris 10 is crashing. I have updated with the latest Splunk 5.0.3 but made it worse. It used generate crash logs and crash files (in /var/core directory) once in every 10 min. Not its twice in 10 min.

Can anyone help ?

Crash log is added below. Looks like the Report Acceleration is causing the crash.


Kind Regards


bash-3.2# more crash-2013-06-27-15:13:19.log
[build 163460] 2013-06-27 15:13:19
Received fatal signal 6 (Abort).
   Unknown signal origin (si_code=-1).
 Crashing thread: dispatch
    RIP:  [0xFFFFFD7FFEAE2CEA] __lwp_kill + 10 (/lib/amd64/libc.so.1)
    RDI:  [0x0000000000000003]
    RSI:  [0x0000000000000006]
    RBP:  [0xFFFFFD7FFE3FD3A0]
    RSP:  [0xFFFFFD7FFE3FD398]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x0000000000000005]
    RDX:  [0xFFFFFFFF83986C80]
    R8:  [0x000000000000002D]
    R9:  [0x0000000000000000]
    R10:  [0x0000000000000005]
    R11:  [0x0000000000000000]
    R12:  [0x0000000002CDB1B8]
    R13:  [0x0000000002CDB010]
    R14:  [0x0000000002CDB1E8]
    R15:  [0x0000000002CB9210]
    RFL:  [0x0000000000000286]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000014]
    CS:  [0x000000000000004B]
    GS:  [0x0000000000000000]
    FS:  [0x0000000000000000]

 OS: SunOS
 Arch: x86-64

  [0xFFFFFD7FFEA87E99] raise + 25 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEA6694E] abort + 94 (/lib/amd64/libc.so.1)
  [0x0000000001A0161F] _ZN9__gnu_cxx27__verbose_terminate_handlerEv + 351 (/opt/splunk/bin/splunkd)
  [0x0000000001A002A6] _ZN10__cxxabiv111__terminateEPFvvE + 6 (/opt/splunk/bin/splunkd)
  [0x0000000001A002D3] _ZSt9terminatev + 19 (/opt/splunk/bin/splunkd)
  [0x0000000001A0065F] __cxa_pure_virtual + 31 (/opt/splunk/bin/splunkd)
  [0x0000000000D1E73E] _ZN15SearchEvaluator10lispyQueryER3StrR7TimevalS3_R9StrVectorRKS2_S7_b + 414 (/opt/splunk/bin/splunkd)
  [0x0000000000B3644C] _ZN17IndexScopedSearch4initERK7TimevalS2_bP14LookupOperatorP12FieldAliaserP18CalcFieldProcessorPKSt3setI10CMBucketIdSt4lessISA_ESaISA_EE + 588 (/opt/splunk/bin/splunkd)
  [0x0000000000B26C9E] _ZN14SearchOperator8evalArgsER17SearchResultsInfo + 9006 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x0000000000B572B9] _ZN22BucketSummaryProcessor8evalArgsER17SearchResultsInfo + 8713 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x000000000103B120] _ZN14DispatchThread8evaluateEbb + 15264 (/opt/splunk/bin/splunkd)
  [0x0000000001033981] _ZN14DispatchThread8mainImplEv + 4321 (/opt/splunk/bin/splunkd)
  [0x00000000010368C2] _ZN14DispatchThread4mainEv + 226 (/opt/splunk/bin/splunkd)
  [0x0000000000F37352] _ZN6Thread8callMainEPv + 98 (/opt/splunk/bin/splunkd)
  [0xFFFFFD7FFEADD1AB] _thr_slot_offset + 795 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEADD3E0] smt_pause + 96 (/lib/amd64/libc.so.1)
 SunOS / splunk / 5.10 / Generic_147441-07 / i86pc
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2013-06-26 17:19:51.400 +1000 splunkd started (build 143156)
    2013-06-26 17:25:11.350 +1000 Interrupt signal received
    2013-06-26 17:27:59.775 +1000 splunkd started (build 143156)
    2013-06-27 12:21:03.153 +1000 Interrupt signal received
    2013-06-27 12:21:56.892 +1000 splunkd started (build 143156)
    2013-06-27 13:21:08.304 +1000 Interrupt signal received
    2013-06-27 13:37:12.340 +1000 splunkd started (build 163460)
    2013-06-27 13:39:12.006 +1000 Interrupt signal received
    2013-06-27 13:39:59.495 +1000 splunkd started (build 163460)
    2013-06-27 13:52:08.211 +1000 Interrupt signal received
    2013-06-27 13:52:58.376 +1000 splunkd started (build 163460)
    2013-06-27 14:50:25.221 +1000 Interrupt signal received
    2013-06-27 15:04:15.911 +1000 splunkd started (build 163460)

Threads running: 3
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=3972] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=3972] search --id=SummaryDirector_1372309985.40 --maxbuckets=0 --ttl=30 --maxout=50000 --maxtime=8640000 --lookups=0 --reduce_freq=10 --user=splunk-system-user --pro --roles=admin:can_delete:cds:power:splunk
0 Karma


Hi KarunK

open files is too low, check the docs about ulimit:

Usually, the default file descriptor limit (ulimit) on a *nix-based OS is 1024. Your Splunk administrator should determine the correct level, but it should be at least 8192.

If this does not help do as kristian told you, make a diag and file a support case.

cheers, MuS


Memory and CPU look good."splukd.log" have a lot of entries like below " DispatchCommand - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/rt_scheduler_nobodycds_RMD5e57c4bb343ae7e10_at_1372658189_0.13518/metadata.csv"

0 Karma


core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 27605
virtual memory (kbytes, -v) unlimited

0 Karma

Ultra Champion

Check your ulimit for open files, and make a diag-dump and open a support case.

0 Karma


Is there anything of interest in $SPLUNK_HOME\var\log\splunk\splunkd.log?

Additionally what's the situation with memory/CPU utilisation?

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...