Splunk Search

Splunk Crashing once in 10min

KarunK
Contributor

Hi All,

My Splunk instance 5.0.1 running in Solaris 10 is crashing. I have updated with the latest Splunk 5.0.3 but made it worse. It used generate crash logs and crash files (in /var/core directory) once in every 10 min. Not its twice in 10 min.

Can anyone help ?

Crash log is added below. Looks like the Report Acceleration is causing the crash.

Thanks

Kind Regards

KK

bash-3.2# more crash-2013-06-27-15:13:19.log
[build 163460] 2013-06-27 15:13:19
Received fatal signal 6 (Abort).
 Cause:
   Unknown signal origin (si_code=-1).
 Crashing thread: dispatch
 Registers:
    RIP:  [0xFFFFFD7FFEAE2CEA] __lwp_kill + 10 (/lib/amd64/libc.so.1)
    RDI:  [0x0000000000000003]
    RSI:  [0x0000000000000006]
    RBP:  [0xFFFFFD7FFE3FD3A0]
    RSP:  [0xFFFFFD7FFE3FD398]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x0000000000000005]
    RDX:  [0xFFFFFFFF83986C80]
    R8:  [0x000000000000002D]
    R9:  [0x0000000000000000]
    R10:  [0x0000000000000005]
    R11:  [0x0000000000000000]
    R12:  [0x0000000002CDB1B8]
    R13:  [0x0000000002CDB010]
    R14:  [0x0000000002CDB1E8]
    R15:  [0x0000000002CB9210]
    RFL:  [0x0000000000000286]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000014]
    CS:  [0x000000000000004B]
    GS:  [0x0000000000000000]
    FS:  [0x0000000000000000]

 OS: SunOS
 Arch: x86-64

 Backtrace:
  [0xFFFFFFFFFFFFFFFF] ?
  [0xFFFFFD7FFEA87E99] raise + 25 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEA6694E] abort + 94 (/lib/amd64/libc.so.1)
  [0x0000000001A0161F] _ZN9__gnu_cxx27__verbose_terminate_handlerEv + 351 (/opt/splunk/bin/splunkd)
  [0x0000000001A002A6] _ZN10__cxxabiv111__terminateEPFvvE + 6 (/opt/splunk/bin/splunkd)
  [0x0000000001A002D3] _ZSt9terminatev + 19 (/opt/splunk/bin/splunkd)
  [0x0000000001A0065F] __cxa_pure_virtual + 31 (/opt/splunk/bin/splunkd)
  [0x0000000000D1E73E] _ZN15SearchEvaluator10lispyQueryER3StrR7TimevalS3_R9StrVectorRKS2_S7_b + 414 (/opt/splunk/bin/splunkd)
  [0x0000000000B3644C] _ZN17IndexScopedSearch4initERK7TimevalS2_bP14LookupOperatorP12FieldAliaserP18CalcFieldProcessorPKSt3setI10CMBucketIdSt4lessISA_ESaISA_EE + 588 (/opt/splunk/bin/splunkd)
  [0x0000000000B26C9E] _ZN14SearchOperator8evalArgsER17SearchResultsInfo + 9006 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x0000000000B572B9] _ZN22BucketSummaryProcessor8evalArgsER17SearchResultsInfo + 8713 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x000000000103B120] _ZN14DispatchThread8evaluateEbb + 15264 (/opt/splunk/bin/splunkd)
  [0x0000000001033981] _ZN14DispatchThread8mainImplEv + 4321 (/opt/splunk/bin/splunkd)
  [0x00000000010368C2] _ZN14DispatchThread4mainEv + 226 (/opt/splunk/bin/splunkd)
  [0x0000000000F37352] _ZN6Thread8callMainEPv + 98 (/opt/splunk/bin/splunkd)
  [0xFFFFFD7FFEADD1AB] _thr_slot_offset + 795 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEADD3E0] smt_pause + 96 (/lib/amd64/libc.so.1)
 SunOS / splunk / 5.10 / Generic_147441-07 / i86pc
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2013-06-26 17:19:51.400 +1000 splunkd started (build 143156)
    2013-06-26 17:25:11.350 +1000 Interrupt signal received
    2013-06-26 17:27:59.775 +1000 splunkd started (build 143156)
    2013-06-27 12:21:03.153 +1000 Interrupt signal received
    2013-06-27 12:21:56.892 +1000 splunkd started (build 143156)
    2013-06-27 13:21:08.304 +1000 Interrupt signal received
    2013-06-27 13:37:12.340 +1000 splunkd started (build 163460)
    2013-06-27 13:39:12.006 +1000 Interrupt signal received
    2013-06-27 13:39:59.495 +1000 splunkd started (build 163460)
    2013-06-27 13:52:08.211 +1000 Interrupt signal received
    2013-06-27 13:52:58.376 +1000 splunkd started (build 163460)
    2013-06-27 14:50:25.221 +1000 Interrupt signal received
    2013-06-27 15:04:15.911 +1000 splunkd started (build 163460)

Threads running: 3
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=3972] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=3972] search --id=SummaryDirector_1372309985.40 --maxbuckets=0 --ttl=30 --maxout=50000 --maxtime=8640000 --lookups=0 --reduce_freq=10 --user=splunk-system-user --pro --roles=admin:can_delete:cds:power:splunk
-system-role:user
terminating...
bash-3.2#
0 Karma

MuS
SplunkTrust
SplunkTrust

Hi KarunK

open files is too low, check the docs about ulimit:

http://docs.splunk.com/Documentation/Splunk/5.0.3/Installation/Systemrequirements
..snip..
Usually, the default file descriptor limit (ulimit) on a *nix-based OS is 1024. Your Splunk administrator should determine the correct level, but it should be at least 8192.
..schnapp..

If this does not help do as kristian told you, make a diag and file a support case.

cheers, MuS

KarunK
Contributor

Memory and CPU look good."splukd.log" have a lot of entries like below " DispatchCommand - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/rt_scheduler_nobodycds_RMD5e57c4bb343ae7e10_at_1372658189_0.13518/metadata.csv"

0 Karma

KarunK
Contributor

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 27605
virtual memory (kbytes, -v) unlimited

0 Karma

kristian_kolb
Ultra Champion

Check your ulimit for open files, and make a diag-dump and open a support case.

0 Karma

MHibbin
Influencer

Is there anything of interest in $SPLUNK_HOME\var\log\splunk\splunkd.log?

Additionally what's the situation with memory/CPU utilisation?

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...