Splunk Search

Splunk Crashing once in 10min

KarunK
Contributor

Hi All,

My Splunk instance 5.0.1 running in Solaris 10 is crashing. I have updated with the latest Splunk 5.0.3 but made it worse. It used generate crash logs and crash files (in /var/core directory) once in every 10 min. Not its twice in 10 min.

Can anyone help ?

Crash log is added below. Looks like the Report Acceleration is causing the crash.

Thanks

Kind Regards

KK

bash-3.2# more crash-2013-06-27-15:13:19.log
[build 163460] 2013-06-27 15:13:19
Received fatal signal 6 (Abort).
 Cause:
   Unknown signal origin (si_code=-1).
 Crashing thread: dispatch
 Registers:
    RIP:  [0xFFFFFD7FFEAE2CEA] __lwp_kill + 10 (/lib/amd64/libc.so.1)
    RDI:  [0x0000000000000003]
    RSI:  [0x0000000000000006]
    RBP:  [0xFFFFFD7FFE3FD3A0]
    RSP:  [0xFFFFFD7FFE3FD398]
    RAX:  [0x0000000000000000]
    RBX:  [0x0000000000000006]
    RCX:  [0x0000000000000005]
    RDX:  [0xFFFFFFFF83986C80]
    R8:  [0x000000000000002D]
    R9:  [0x0000000000000000]
    R10:  [0x0000000000000005]
    R11:  [0x0000000000000000]
    R12:  [0x0000000002CDB1B8]
    R13:  [0x0000000002CDB010]
    R14:  [0x0000000002CDB1E8]
    R15:  [0x0000000002CB9210]
    RFL:  [0x0000000000000286]
    TRAPNO:  [0x000000000000000E]
    ERR:  [0x0000000000000014]
    CS:  [0x000000000000004B]
    GS:  [0x0000000000000000]
    FS:  [0x0000000000000000]

 OS: SunOS
 Arch: x86-64

 Backtrace:
  [0xFFFFFFFFFFFFFFFF] ?
  [0xFFFFFD7FFEA87E99] raise + 25 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEA6694E] abort + 94 (/lib/amd64/libc.so.1)
  [0x0000000001A0161F] _ZN9__gnu_cxx27__verbose_terminate_handlerEv + 351 (/opt/splunk/bin/splunkd)
  [0x0000000001A002A6] _ZN10__cxxabiv111__terminateEPFvvE + 6 (/opt/splunk/bin/splunkd)
  [0x0000000001A002D3] _ZSt9terminatev + 19 (/opt/splunk/bin/splunkd)
  [0x0000000001A0065F] __cxa_pure_virtual + 31 (/opt/splunk/bin/splunkd)
  [0x0000000000D1E73E] _ZN15SearchEvaluator10lispyQueryER3StrR7TimevalS3_R9StrVectorRKS2_S7_b + 414 (/opt/splunk/bin/splunkd)
  [0x0000000000B3644C] _ZN17IndexScopedSearch4initERK7TimevalS2_bP14LookupOperatorP12FieldAliaserP18CalcFieldProcessorPKSt3setI10CMBucketIdSt4lessISA_ESaISA_EE + 588 (/opt/splunk/bin/splunkd)
  [0x0000000000B26C9E] _ZN14SearchOperator8evalArgsER17SearchResultsInfo + 9006 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x0000000000B572B9] _ZN22BucketSummaryProcessor8evalArgsER17SearchResultsInfo + 8713 (/opt/splunk/bin/splunkd)
  [0x0000000000FA901A] _ZN14SearchPipeline8evalArgsER17SearchResultsInfo + 90 (/opt/splunk/bin/splunkd)
  [0x000000000103B120] _ZN14DispatchThread8evaluateEbb + 15264 (/opt/splunk/bin/splunkd)
  [0x0000000001033981] _ZN14DispatchThread8mainImplEv + 4321 (/opt/splunk/bin/splunkd)
  [0x00000000010368C2] _ZN14DispatchThread4mainEv + 226 (/opt/splunk/bin/splunkd)
  [0x0000000000F37352] _ZN6Thread8callMainEPv + 98 (/opt/splunk/bin/splunkd)
  [0xFFFFFD7FFEADD1AB] _thr_slot_offset + 795 (/lib/amd64/libc.so.1)
  [0xFFFFFD7FFEADD3E0] smt_pause + 96 (/lib/amd64/libc.so.1)
 SunOS / splunk / 5.10 / Generic_147441-07 / i86pc
 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2013-06-26 17:19:51.400 +1000 splunkd started (build 143156)
    2013-06-26 17:25:11.350 +1000 Interrupt signal received
    2013-06-26 17:27:59.775 +1000 splunkd started (build 143156)
    2013-06-27 12:21:03.153 +1000 Interrupt signal received
    2013-06-27 12:21:56.892 +1000 splunkd started (build 143156)
    2013-06-27 13:21:08.304 +1000 Interrupt signal received
    2013-06-27 13:37:12.340 +1000 splunkd started (build 163460)
    2013-06-27 13:39:12.006 +1000 Interrupt signal received
    2013-06-27 13:39:59.495 +1000 splunkd started (build 163460)
    2013-06-27 13:52:08.211 +1000 Interrupt signal received
    2013-06-27 13:52:58.376 +1000 splunkd started (build 163460)
    2013-06-27 14:50:25.221 +1000 Interrupt signal received
    2013-06-27 15:04:15.911 +1000 splunkd started (build 163460)

Threads running: 3
argv: [splunkd -p 8089 start]
Process renamed: [splunkd pid=3972] splunkd -p 8089 start [process-runner]
Process renamed: [splunkd pid=3972] search --id=SummaryDirector_1372309985.40 --maxbuckets=0 --ttl=30 --maxout=50000 --maxtime=8640000 --lookups=0 --reduce_freq=10 --user=splunk-system-user --pro --roles=admin:can_delete:cds:power:splunk
-system-role:user
terminating...
bash-3.2#
0 Karma

MuS
Legend

Hi KarunK

open files is too low, check the docs about ulimit:

http://docs.splunk.com/Documentation/Splunk/5.0.3/Installation/Systemrequirements
..snip..
Usually, the default file descriptor limit (ulimit) on a *nix-based OS is 1024. Your Splunk administrator should determine the correct level, but it should be at least 8192.
..schnapp..

If this does not help do as kristian told you, make a diag and file a support case.

cheers, MuS

KarunK
Contributor

Memory and CPU look good."splukd.log" have a lot of entries like below " DispatchCommand - could not read metadata file: /opt/splunk/var/run/splunk/dispatch/rt_scheduler_nobodycds_RMD5e57c4bb343ae7e10_at_1372658189_0.13518/metadata.csv"

0 Karma

KarunK
Contributor

core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
file size (blocks, -f) unlimited
open files (-n) 256
pipe size (512 bytes, -p) 10
stack size (kbytes, -s) 10240
cpu time (seconds, -t) unlimited
max user processes (-u) 27605
virtual memory (kbytes, -v) unlimited

0 Karma

kristian_kolb
Ultra Champion

Check your ulimit for open files, and make a diag-dump and open a support case.

0 Karma

MHibbin
Influencer

Is there anything of interest in $SPLUNK_HOME\var\log\splunk\splunkd.log?

Additionally what's the situation with memory/CPU utilisation?

0 Karma
Get Updates on the Splunk Community!

Index This | Divide 100 by half. What do you get?

November 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with this ...

Stay Connected: Your Guide to December Tech Talks, Office Hours, and Webinars!

❄️ Celebrate the season with our December lineup of Community Office Hours, Tech Talks, and Webinars! ...

Splunk and Fraud

Watch Now!Watch an insightful webinar where we delve into the innovative approaches to solving fraud using the ...