Splunk Search

DMP files are killing my drive!!

ethompso
Explorer

Every 10 min DMP files and the text document are being created on my drive:

C__Program Files_Splunk_bin_splunkd_exe_crash-2013-11-16-14-30-31.dmp
C__Program Files_Splunk_bin_splunkd_exe_crash-2013-11-16-14-30-31

Why is that and how can I fix whatever the issue is?

Thank you for your assistance.

    [build 182037] 2013-11-16 14:30:31
 Access violation, cannot read at address [0x000001000006B425]
 Exception address: [0x00007FF6AE335B96]
 Crashing thread: dispatch
    MxCsr:  [0x0000000000001FA0]
    SegDs:  [0x000000000000002B]
    SegEs:  [0x000000000000002B]
    SegFs:  [0x0000000000000053]
    SegGs:  [0x000000000000002B]
    SegSs:  [0x000000000000002B]
    SegCs:  [0x0000000000000033]
    EFlags:  [0x0000000000010286]
    Rsp:  [0x00000093B7E0BD40]
    Rip:  [0x00007FF6AE335B96] ?
    Dr0:  [0x0000000000000000]
    Dr1:  [0x0000000000000000]
    Dr2:  [0x0000000000000000]
    Dr3:  [0x0000000000000000]
    Dr6:  [0x0000000000000000]
    Dr7:  [0x0000000000000000]
    Rax:  [0x000001000006B425]
    Rcx:  [0x00000093B85E3980]
    Rdx:  [0x00000093B7E0BD70]
    Rbx:  [0x00000093B85E5050]
    Rbp:  [0x00000093B7E0BDF1]
    Rsi:  [0x00000093B85E3980]
    Rdi:  [0x0000000000000000]
    R8:  [0x00000093B7302870]
    R9:  [0x0000000000000100]
    R10:  [0x0000000000000100]
    R11:  [0x0000000000000004]
    R12:  [0x00000093B7E0BF48]
    R13:  [0x00000093B861E130]
    R14:  [0x00000093B85E4DD0]
    R15:  [0x00000093B85E3700]
    DebugControl:  [0x00007FF6ADFF6A71]
    LastBranchToRip:  [0x0000000000000000]
    LastBranchFromRip:  [0x0000000000000000]
    LastExceptionToRip:  [0x0000000000000000]
    LastExceptionFromRip:  [0x0000000000000000]

 OS: Windows
 Arch: x86-64

 Backtrace:
  [0x00007FF6AE335B96] ?
  [0x00007FF6AE1B321F] ?
  [0x00007FF6AE13B40B] ?
  [0x00007FF6AE88B2A4] ?
  [0x00007FF6AE175E11] ?
  [0x00007FF6AE88B2A4] ?
  [0x00007FF6AEA29CB3] ?
  [0x00007FF6AEA274F8] ?
  [0x00007FF6AEA2317E] ?
  [0x00007FF6AEA19368] ?
  [0x00007FF6AEA15B35] ?
  [0x00007FF6AE8AD10E] ?
  [0x00007FF6AE8A882A] ?
  [0x00007FF6AE8A7D89] ?
  [0x00007FF6AE53720E] ?
  [0x00007FF6ADE69537] ?
  [0x00007FFB1DCE3FEF] beginthreadex + 263/284
  [0x00007FFB1DCE4196] endthreadex + 402/404
  [0x00007FFB270515BD] BaseThreadInitThunk + 13/48
  [0x00007FFB297843D1] RtlUserThreadStart + 29/68
 Crash dump written to: C:\Program Files\Splunk\var\log\splunk\C__Program Files_Splunk_bin_splunkd_exe_crash-2013-11-16-14-30-31.dmp

MY-LAPTOP /6.2 
GetLastError(): 0
Threads running: 6
argv: [splunkd search --id=scheduler__admin__ipreputation__RMD5336336c7fbf5268b_at_1384630200_3 --maxbuckets=0 --ttl=60 --maxout=500000 --maxtime=8640000 --lookups=1 --reduce_freq=10 --user=admin --pro --roles=admin:power:user]
Thread: "dispatch", did_join=1, ready_to_run=Y, main_thread=N
First 4 bytes of Thread token @00000093B79E8EBC:
00000000  90 38 00 00                                       |.8..|
00000004

x86 CPUID registers:
         0: 0000000D 756E6547 6C65746E 49656E69
         1: 000306A9 03100800 7FBAE3BF BFEBFBFF
         2: 76035A01 00F0B2FF 00000000 00CA0000
         3: 00000000 00000000 00000000 00000000
         4: 1C004121 01C0003F 0000003F 00000000
         5: 00000040 00000040 00000003 00021120
         6: 00000077 00000002 00000009 00000000
         7: 00000000 00000281 00000000 00000000
         8: 00000000 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 07300403 00000000 00000000 00000603
         B: 00000001 00000002 00000100 00000003
         C: 00000000 00000000 00000000 00000000
         😧 00000007 00000340 00000340 00000000
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 28100800
  80000002: 20202020 49202020 6C65746E 20295228
  80000003: 65726F43 294D5428 2D356920 30333233
  80000004: 5043204D 20402055 30362E32 007A4847
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 01006040 00000000
  80000007: 00000000 00000000 00000000 00000100
  80000008: 00003024 00000000 00000000 00000000
terminating...
Tags (1)

jichen
Explorer

hi, I have the same problem.I run splunk version 5.0.5 on win2008R with a 16 cores cpu and 32GB ram. The dump file generated per 10 mins. Later I'll post the detail.

0 Karma

lguinn2
Legend

Instead of trying to read the .dmp files, which probably none of us can do, I would examine splunkd.log

What was Splunk doing just before the crash?

Also, does your laptop meet the minimum hardware requirements for Splunk?

My guess is that Splunk unexpectedly ran out of resource while launching a search.

What version of Splunk are you running? On what platform?

0 Karma

ethompso
Explorer

Is anyone else experiencing the same? Just checked another machine the same is happening, but not as often. Once a day compared to every 10 minutes.

0 Karma

ethompso
Explorer

Will do. Thanks

0 Karma

rsennett_splunk
Splunk Employee
Splunk Employee

You probably want to open a support case...

With Splunk... the answer is always "YES!". It just might require more regex than you're prepared for!
0 Karma