Monitoring Splunk

Failures after Upgrading to 6.0

ShaneNewman
Motivator

We upgraded to 6.0 yesterday. Since then, randomly, we see this:

    [build 182037] 2013-11-26 15:01:46
     Access violation, cannot read at address [0x0000000014F00000]
     Exception address: [0x000000000037C5F4]
     Crashing thread: HTTPDispatch
        MxCsr:  [0x0000000000001F80]
        SegDs:  [0x000000000000002B]
        SegEs:  [0x000000000000002B]
        SegFs:  [0x0000000000000053]
        SegGs:  [0x000000000000002B]
        SegSs:  [0x000000000000002B]
        SegCs:  [0x0000000000000033]
        EFlags:  [0x0000000000010206]
        Rsp:  [0x000000000287FA68]
        Rip:  [0x000000000037C5F4] memmove + 628/1280
        Dr0:  [0x0000000000000000]
        Dr1:  [0x0000000000000000]
        Dr2:  [0x0000000000000000]
        Dr3:  [0x0000000000000000]
        Dr6:  [0x0000000000000000]
        Dr7:  [0x0000000000000000]
        Rax:  [0x000000000000000B]
        Rcx:  [0x000000000BFE0518]
        Rdx:  [0x0000000008F1FAE0]
        Rbx:  [0x00000000104FD3D8]
        Rbp:  [0x0000000000003FD8]
        Rsi:  [0x000000000BFE0038]
        Rdi:  [0x0000000000003FD8]
        R8:  [0x0000000000003FD8]
        R9:  [0x00000000000001D7]
        R10:  [0x000000000000000B]
        R11:  [0x000000000BFE0038]
        R12:  [0x0000000000000002]
        R13:  [0x0000000000000002]
        R14:  [0x0000000000000002]
        R15:  [0x0000000000000000]
        DebugControl:  [0xFFFFF80001024D4A]
        LastBranchToRip:  [0x0000000000000000]
        LastBranchFromRip:  [0x0000000000000000]
        LastExceptionToRip:  [0x0000000000000000]
        LastExceptionFromRip:  [0x0000000000000000]

     OS: Windows
     Arch: x86-64

Backtrace:
  [0x000000000037C5F4] memmove + 628/1280
  [0x000000000FBD0371] FINGERPRINT_premain + 10145/33104
  [0x000000000FBCFEC7] FINGERPRINT_premain + 8951/33104
  [0x000000000FBCE4E5] FINGERPRINT_premain + 2325/33104
  [0x000000000FBCF175] FINGERPRINT_premain + 5541/33104
  [0x000000000FBB83E5] COMP_rle + 709/2272
  [0x000000000FBB7FC7] COMP_compress_block + 55/80
  [0x000000000048468A] SSLv3_client_method + 38346/47744
  [0x0000000000480EC7] SSLv3_client_method + 24071/47744
  [0x000000014079D9C9] ?
  [0x00000001407A0C0C] ?
  [0x00000001407A091F] ?
  [0x000000014079CADB] ?
  [0x0000000140026D2C] ?
  [0x00000001406F720E] ?
  [0x0000000140029537] ?
  [0x0000000000363FEF] beginthreadex + 263/284
  [0x0000000000364196] endthreadex + 402/404
  [0x0000000077D6B70A] BaseThreadStart + 58/80
 Crash dump written to: C:\Program Files\Splunk\var\log\splunk\C__Program Files_Splunk_bin_splunkd_exe_crash-2013-11-26-15-01-46.dmp

TRDVWPTRM01 /5.2 Service Pack 2
GetLastError(): 0
Threads running: 113
argv: [Splunkd -p 8089]
Thread: "HTTPDispatch", did_join=0, ready_to_run=Y, main_thread=N
First 4 bytes of Thread token @00000000023BD344:
00000000  88 07 00 00                                       |....|
00000004
First 128 bytes of CompletionPortItem object @000000001203F560:
00000000  18 af 15 41 01 00 00 00  50 b7 34 25 00 00 00 00  |...A....P.4%....|
00000010  38 96 26 41 01 00 00 00  01 00 00 00 00 00 00 00  |8.&A............|
00000020  03 01 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000030  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000040  00 00 00 00 00 00 00 00  00 10 00 00 00 00 00 00  |................|
00000050  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  |................|
00000060  40 97 da 11 00 00 00 00  00 00 00 00 00 00 00 00  |@...............|
00000070  00 00 00 00 00 00 00 00  47 45 54 20 2f 73 65 72  |........GET /ser|
00000080
In TcpChannel 000000001203F560, _tcloop=000000000166FC18, no async write data, _data._shouldKill=N, r/w_timeouts=5000/300000, timeout_count=0
SSL: version="SSLv3", state="SSL negotiation finished successfully", cipher="AES256-SHA", compression="zlib compression"
rbuf: ptr=000000001203F5D8, size=0x2000, rptr=0x0, wptr=0x0
TcpChannelAcceptor: pendingAsyncAccept=1, dyingChannels=0, tcloop=000000000166FC18, activeList=000000001F34F230, activeCount=10, idleThreads=0000000013CCF058, idleThreadCount=43, activeThreadCount=9, maxActiveThreadsInTimePeriod=27, idleThreadTrimmerActive=Y, disabledReasons=0
_thread=000000001401B958: commandForThread=0, nextIdle=0000000011FD4370, requestAfterThread=2, _tpfd=000000001203F560writeCorkCount=1, terminateCallback=0000000000000000, ioError=No error, lastError=No error, terminateError=Connection closed by peer
giveCmd @000000001401BA78: _queuedOn=0000000000000000, ran=N, wantWake=N, wantFailIfLoopDone=N, cmd=2, ok=Y, chan=000000001203F560
writeDataAvail @000000001401BAB0: _queuedOn=0000000000000000, ran=N, wantWake=N, wantFailIfLoopDone=N, chan=000000001203F560
wbuf: ptr=000000001401BB18, size=0x8000, rptr=0x0, wptr=0x8000


x86 CPUID registers:
         0: 0000000A 756E6547 6C65746E 49656E69
         1: 000006FB 09040800 0004E3BD BFEBFBFF
         2: 05B0B101 005657F0 00000000 2CB43048
         3: 00000000 00000000 00000000 00000000
         4: 0C000121 01C0003F 0000003F 00000001
         5: 00000040 00000040 00000003 00002220
         6: 00000001 00000002 00000001 00000000
         7: 00000000 00000000 00000000 00000000
         8: 00000400 00000000 00000000 00000000
         9: 00000000 00000000 00000000 00000000
         A: 07280202 00000000 00000000 00000503
  80000000: 80000008 00000000 00000000 00000000
  80000001: 00000000 00000000 00000001 20100800
  80000002: 65746E49 2952286C 6F655820 2952286E
  80000003: 55504320 20202020 20202020 45202020
  80000004: 30333337 20402020 30342E32 007A4847
  80000005: 00000000 00000000 00000000 00000000
  80000006: 00000000 00000000 0C007040 00000000
  80000007: 00000000 00000000 00000000 00000000
  80000008: 00003028 00000000 00000000 00000000
terminating...

Splunk crashes, a restart gets it working for a while, then it crashes again. Any ideas what I need to do in order to correct this?

Tags (3)
0 Karma
1 Solution

jordanperks
Path Finder

There are people here much smarter than me, but I think "argv: [Splunkd -p 8089]" is the source of the crash. 8089 is default for listening on the splunkd management port. There is some more information about it here: The disableDefaultPort = [true|false] setting is documented here: http://www.splunk.com/base/Documentation/latest/admin/Serverconf. I found that in this thread http://answers.splunk.com/answers/26236/universal-forwarder-listening-on-port-8089

View solution in original post

jordanperks
Path Finder

There are people here much smarter than me, but I think "argv: [Splunkd -p 8089]" is the source of the crash. 8089 is default for listening on the splunkd management port. There is some more information about it here: The disableDefaultPort = [true|false] setting is documented here: http://www.splunk.com/base/Documentation/latest/admin/Serverconf. I found that in this thread http://answers.splunk.com/answers/26236/universal-forwarder-listening-on-port-8089

ShaneNewman
Motivator

I created a critical ticket with Splunk. Turns out we needed to deploy the pre-release of Splunk. The issue then, was that Splunk 6.0.0.2 does not work on WS2003R2 and a bug has been filed.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...