Splunk Crashes After Login - After Upgrade to 7.2.5

[StolenEclipse]

Hello,

Following the upgrade to Splunk 7.2.5 yesterday my Splunk (single instance, Windows) server will not progress after logging in.

After entering my credentials and attempting to click on anything in the UI, the server stops responding, Splunkd service stops and a dump file is generated.

This is the latest one:

[build 088f49762779] 2019-03-22 11:20:07
Access violation, cannot read at address [0xFFFFFFFFFFFFFFFF]
Exception address: [0x00007FF652F85CDA]
Crashing thread: SchedulerThread
MxCsr: [0x0000000000001FA0]
SegDs: [0x000000000000002B]
SegEs: [0x000000000000002B]
SegFs: [0x0000000000000053]
SegGs: [0x000000000000002B]
SegSs: [0x000000000000002B]
SegCs: [0x0000000000000033]
EFlags: [0x0000000000010202]
Rsp: [0x00000052F3EA9400]
Rip: [0x00007FF652F85CDA] ?
Dr0: [0x0000000000000000]
Dr1: [0x0000000000000000]
Dr2: [0x0000000000000000]
Dr3: [0x0000000000000000]
Dr6: [0x0000000000000000]
Dr7: [0x0000000000000000]
Rax: [0x00000052EFDEF0A0]
Rcx: [0x6C757365722D5341]
Rdx: [0x00000052F3EA96E0]
Rbx: [0x00000052FE6313E0]
Rbp: [0x00000052F3EA9500]
Rsi: [0x00000052FE9808B0]
Rdi: [0x00000052F3EA96E0]
R8: [0x00000000000002A7]
R9: [0x0000000000000033]
R10: [0x00000052EFDEC090]
R11: [0x0000000000000000]
R12: [0x0000000000000000]
R13: [0x00000052F8EE7560]
R14: [0x00000052E7E12630]
R15: [0x0000000000000000]
DebugControl: [0x00007FF6523E84DA]
LastBranchToRip: [0x0000000000000000]
LastBranchFromRip: [0x0000000000000000]
LastExceptionToRip: [0x0000000000000000]
LastExceptionFromRip: [0x0000000000000000]

OS: Windows
Arch: x86-64

Backtrace:
[0x00007FF652F85CDA] ?
Args: [0x0000000000000000] [0x00000052F8EE7560] [0x00000052FE980890]
[0x00007FF652F86048] ?
Args: [0x00000052F8EE7560] [0x0000005288C1D8F0] [0x0000000000000024]
[0x00007FF652F88257] ?
Args: [0x00000052F8EE7560] [0x00000052F61B0BE0] [0x00000052F8EE7560]
[0x00007FF6523E211D] ?
Args: [0x0000000000000000] [0x00007FF652A4D324] [0x00000052FE9A2010]
[0x00007FF6523E1403] ?
Args: [0x0000000000000000] [0x00000052FE9A2010] [0x0000000000000000]
[0x00007FF652EF0541] ?
Args: [0x0000000000000000] [0x0000000000000000] [0x00000052F3EAD428]
[0x00007FF652FCDA69] ?
Args: [0x0000000000000000] [0x00000052F3EAD650] [0x00000052F3EAD6F0]
[0x00007FF6524E5FDE] ?
Args: [0x00000052F3EAD6F0] [0x0000000000000000] [0x0000000000000000]
[0x00007FF6524EC334] ?
Args: [0x0000000000000001] [0x00000052FE4FAF00] [0x00000052FE4FB440]
[0x00007FF6524EBE04] ?
Args: [0x0000000000000000] [0x0000005285766420] [0x0000005285766500]
[0x00007FF6524FE4BF] ?
Args: [0x00000052F60C9770] [0x00000052F3EAEBD0] [0x0000000000000012]
[0x00007FF65250F62C] ?
Args: [0x00000052F3EAE950] [0x0000005281FA52E0] [0x0000005281FA52E0]
[0x00007FF65251DA90] ?
Args: [0x00000052FC157AE0] [0x00007FF6539077A0] [0x000000005C950B35]
[0x00007FF652518673] ?
Args: [0x00007FF653A11B88] [0x00000052F60C9770] [0x00000052F60C9778]
[0x00007FF65252CE71] ?
Args: [0x00000052F60C9770] [0x00000052F3EAEE00] [0x00000052FAAF0C10]
[0x00007FF652A19CFF] ?
Args: [0x0000000000000000] [0x00000000008C37CE] [0x0000000000007530]
[0x00007FF652A1D6CF] ?
Args: [0x00007FF6542EA100] [0x00000000008C3724] [0x0000000000000000]
[0x00007FF652523732] ?
Args: [0x00000052E7DF6900] [0x00000052E7DF6900] [0x00000052E7DE4E20]
[0x00007FF652A04760] ?
Args: [0x0000000000000000] [0x00007FF9661FBDC0] [0x00007FF9661FBDC0]
[0x00007FF9661FBE1D] crt_at_quick_exit + 125/784
Args: [0x00007FF9661FBDC0] [0x00000052E7DE4E20] [0x0000000000000000]
[0x00007FF9794113D2] BaseThreadInitThunk + 34/96
Args: [0x00007FF9794113B0] [0x0000000000000000] [0x0000000000000000]
[0x00007FF97BBD54F4] RtlUserThreadStart + 52/1008
Args: [0x0000000000000000] [0x0000000000000000] [0x0000000000000000]
Crash dump written to: C:\Program Files\Splunk\var\log\splunk\C__Program Files_Splunk_bin_splunkd_exe_crash-2019-03-22-11-20-07.dmp

Splunk ran as local administrator
SPLUNK /Windows Server 2012 R2
GetLastError(): 0
Threads running: 98
Executable module base: 0x00007FF651F20000
Runtime: 53.840754s
argv: [Splunkd -p 8089]
Thread: "SchedulerThread", did_join=0, ready_to_run=Y, main_thread=N
First 4 bytes of Thread token @0x52e7df6914:
00000000 58 02 00 00 |X...|
00000004

First 512 bytes of Timeout object @0x52f60c9770:
00000000 50 6f 90 53 f6 7f 00 00 a0 f0 ea f3 52 00 00 00 |Po.S........R...|
00000010 70 ed ea f3 52 00 00 00 00 00 00 00 00 00 00 00 |p...R...........|
00000020 00 00 00 00 00 00 00 00 14 37 8c 00 00 00 00 00 |.........7......|
00000030 30 75 00 00 00 00 00 00 d8 dd 32 54 f6 7f 00 00 |0u........2T....|
00000040 40 37 2e 54 f6 7f 00 00 00 1e 24 f6 52 00 00 00 |@7.T......$.R...|
00000050 00 00 64 e6 52 00 00 00 10 00 00 00 00 00 00 00 |..d.R...........|
00000060 1f 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000070 01 00 64 e6 52 00 00 00 ff ff ff ff ff ff ff ff |..d.R...........|
00000080 ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000090 00 00 00 00 00 00 00 00 d0 07 00 02 00 00 00 00 |................|
000000a0 78 6f 90 53 f6 7f 00 00 90 99 b3 e8 52 00 00 00 |xo.S........R...|
000000b0 00 00 00 00 00 00 00 00 aa 74 6a 9c ac ea df a9 |.........tj.....|
000000c0 03 00 00 00 00 00 00 00 a0 2e e7 f5 52 00 00 00 |............R...|
000000d0 a0 75 b0 e8 52 00 00 00 46 00 00 00 00 00 00 00 |.u..R...F.......|
000000e0 ff ff ff ff ff ff ff ff ff ff ff ff 00 00 00 00 |................|
000000f0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000100 d0 07 00 02 00 00 00 00 70 d3 af e8 52 00 00 00 |........p...R...|
00000110 00 00 00 00 00 00 00 00 30 b2 97 e9 52 00 00 00 |........0...R...|
00000120 00 00 00 00 00 00 00 00 ff ff ff ff ff ff ff ff |................|
00000130 ff ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000140 00 00 00 00 00 00 00 00 d0 07 00 02 00 00 00 00 |................|
00000150 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000160 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000170 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000180 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000190 00 00 00 00 00 00 00 00 01 00 ca e9 52 00 00 00 |............R...|
000001a0 50 9a 30 f5 52 00 00 00 c0 70 fc f5 52 00 00 00 |P.0.R....p..R...|
000001b0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001c0 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001d0 90 99 30 f5 52 00 00 00 30 6f fc f5 52 00 00 00 |..0.R...0o..R...|
000001e0 08 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000001f0 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000200
_when = 9189.140, _initialInterval = 30.000

x86 CPUID registers:
0: 0000000D 756E6547 6C65746E 49656E69
1: 00050654 03040800 FFFA3203 1FABFBFF
2: 76036301 00F0B5FF 00000000 00C30000
3: 00000000 00000000 00000000 00000000
4: 0C00C121 01C0003F 0000003F 00000000
5: 00000000 00000000 00000000 00000000
6: 00000004 00000000 00000000 00000000
7: 00000000 D19F6FFB 00000008 2C000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300401 0000007F 00000000 00000000
B: 00000000 00000001 00000100 00000003
C: 00000000 00000000 00000000 00000000
😧 000002FF 00000340 00000A88 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000121 2C100800
80000002: 65746E49 2952286C 6F655820 2952286E
80000003: 6C6F4720 31362064 43203033 40205550
80000004: 312E3220 7A484730 00000000 00000000
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 0000302B 00000000 00000000 00000000
terminating...

Any help is appreciated!

[StolenEclipse]

Thanks for your responses. I did make a ticket with Splunk, but I ended up restoring the server.

Thanks again!

[andreasz]

On SUSE Linux Splunk 7.2.5 crashes too:

[build 088f49762779] 2019-03-25 11:19:56
Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 4302 running under UID 0.
Crashing thread: Main Thread
Registers:
RIP: [0x00007FA7BA97BF67] gsignal + 55 (libc.so.6 + 0x34F67)
RDI: [0x00000000000010CE]
RSI: [0x00000000000010CE]
RBP: [0x00007FA7BACE76D8]
RSP: [0x00007FFF638DDE18]
RAX: [0x0000000000000000]
RBX: [0x00007FA7BA11ACE0]
RCX: [0x00007FA7BA97BF67]
RDX: [0x0000000000000006]
R8: [0x000000000000000A]
R9: [0x00007FA7BBE387C0]
R10: [0x0000000000000008]
R11: [0x0000000000000202]
R12: [0x00007FA7BA4C56A0]
R13: [0x00007FA7BA11AC90]
R14: [0x0000562779FDC920]
R15: [0x0000000000000000]
EFL: [0x0000000000000202]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
[0x00007FA7BA97BF67] gsignal + 55 (libc.so.6 + 0x34F67)
[0x00007FA7BA97D33A] abort + 314 (libc.so.6 + 0x3633A)
[0x00005627796467AD] ZN9gnu_cxx27verbose_terminate_handlerEv + 349 (search-launcher + 0x25107AD)
[0x00005627795CC1D6] _ZN10cxxabiv111_terminateEPFvvE + 6 (search-launcher + 0x24961D6)
[0x00005627795CC221] ? (search-launcher + 0x2496221)
[0x00005627795CC5E8] ? (search-launcher + 0x24965E8)
[0x0000562777BA78B9] ? (search-launcher + 0xA718B9)
[0x0000562778816BD8] _Z17ProcessRunnerInitRK8Pathname + 808 (search-launcher + 0x16E0BD8)
[0x0000562778E933AA] ? (search-launcher + 0x1D5D3AA)
[0x0000562778817253] _ZN32ProcessRunnerChildCommandHandler19handle_command_forkEPK18proto_fork_commandPKv + 803 (search-launcher + 0x16E1253)
[0x00005627788175D4] _ZN32ProcessRunnerChildCommandHandler11consumeDataERK3Str + 356 (search-launcher + 0x16E15D4)
[0x000056277879494B] _ZN21SocketCommandConsumer13when_readableE18PollableDescriptor + 219 (search-launcher + 0x165E94B)
[0x0000562778795D25] _ZN12PolledSocket3Pfd11when_eventsE18PollableDescriptor + 85 (search-launcher + 0x165FD25)
[0x0000562778796246] _ZN8PolledFd8do_eventEv + 134 (search-launcher + 0x1660246)
[0x000056277879721B] _ZN9EventLoop3runEv + 651 (search-launcher + 0x166121B)
[0x00005627788166AC] _ZN20ExternalProcessGroup20process_runner_childERK8PathnameR14PolledReadPipe + 1084 (search-launcher + 0x16E06AC)
[0x0000562778816BC5] _Z17ProcessRunnerInitRK8Pathname + 789 (search-launcher + 0x16E0BC5)
[0x00005627782FBBD8] _ZN16SplunkMainThreadC1ERK8Pathnameb + 1064 (search-launcher + 0x11C5BD8)
[0x0000562777BBE7D1] main + 17937 (search-launcher + 0xA887D1)
[0x00007FA7BA967725] __libc_start_main + 245 (libc.so.6 + 0x20725)
[0x0000562777C909A2] ? (search-launcher + 0xB5A9A2)
Linux / zbm-sl-muc-0099 / 4.4.155-94.50-default / #1 SMP Tue Sep 11 13:04:00 UTC 2018 (bc8c7c0) / x86_64
/etc/SuSE-release: SUSE Linux Enterprise Server 12 (x86_64)
glibc version: 2.22
glibc release: stable
Last errno: 11
Threads running: 1
Runtime: 176.912315s
argv: [splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd]
Process renamed: [splunkd pid=1567] splunkd --under-systemd --systemd-delegate=yes -p 8089 _internal_launch_under_systemd [process-runner]
Process renamed: [splunkd pid=1567] [search-launcher]

Regex JIT enabled

using CLOCK_MONOTONIC
Preforked process=0/20: process_runtime_msec=12, completed_searches=0, user_changes=0, cache_rotations=0

First 512 bytes of PolledFd object @0x7fff638dec90:
00000000 e0 8c fd 79 27 56 00 00 00 00 00 00 da dd 42 65 |...y'V........Be|
00000010 00 8f 08 7a 27 56 00 00 00 00 00 00 00 00 00 00 |...z'V..........|
00000020 80 ef 8d 63 ff 7f 00 00 00 00 00 00 00 00 00 00 |...c............|
00000030 00 00 00 00 00 00 f0 3f 08 00 00 00 03 00 00 00 |.......?........|
00000040 40 b5 0b ba a7 7f 00 00 1b 20 00 00 ff 7f 00 00 |@........ ......|
00000050 00 00 00 00 00 00 00 00 7f 03 8d 63 ff 7f 00 00 |...........c....|
00000060 80 ec 8d 63 ff 7f 00 00 08 ed 8d 63 ff 7f 00 00 |...c.......c....|
00000070 00 00 00 00 00 00 00 00 00 f2 8d 63 ff 7f 00 00 |...........c....|
00000080 01 00 00 00 00 00 00 00 70 ee 8d 63 ff 7f 00 00 |........p..c....|
00000090 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000a0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
000000e0 01 00 00 00 00 00 00 00 00 a0 12 ba a7 7f 00 00 |................|
000000f0 65 00 00 00 00 00 00 00 30 0b 00 00 00 00 00 00 |e.......0.......|
00000100 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000110 00 00 00 00 00 00 00 00 70 8d 12 ba a7 7f 00 00 |........p.......|
00000120 d0 87 12 ba a7 7f 00 00 30 9a 12 ba a7 7f 00 00 |........0.......|
00000130 12 00 00 00 00 00 00 00 08 7e 02 bc a7 7f 00 00 |.........~......|
00000140 00 00 00 00 ff 7f 00 00 e0 8c 12 ba a7 7f 00 00 |................|
00000150 60 88 12 ba a7 7f 00 00 a0 99 12 ba a7 7f 00 00 |`...............|
00000160 12 00 00 00 00 00 00 00 00 ec 8d 63 ff 7f 00 00 |...........c....|
00000170 70 ee 8d 63 ff 7f 00 00 b0 07 16 ba a7 7f 00 00 |p..c............|
00000180 26 00 00 00 00 00 00 00 26 00 00 00 00 00 00 00 |&.......&.......|
00000190 00 00 00 00 00 00 00 00 10 5d 0a ba a7 7f 00 00 |.........]......|
000001a0 2a 00 00 00 00 00 00 00 2a 00 00 00 00 00 00 00 |..............|
000001b0 00 00 00 00 00 00 00 00 1f 06 00 00 54 07 00 00 |............T...|
000001c0 d0 e5 4e ba a7 7f 00 00 d0 e5 4e ba a7 7f 00 00 |..N.......N.....|
000001d0 e0 e5 4e ba a7 7f 00 00 00 00 00 00 00 00 00 00 |..N.............|
000001e0 fe 22 b8 39 c1 c8 57 3f c6 00 00 00 00 00 00 00 |.".9..W?........|
000001f0 af 71 59 03 00 00 00 00 01 70 72 6f 63 2f 73 65 |.qY......proc/se|
00000200

x86 CPUID registers:
0: 0000000D 756E6547 6C65746E 49656E69
1: 000406F0 0E010800 FFFA3203 0FABFBFF
2: 76036301 00F0B5FF 00000000 00C30000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000000 00000000 00000000 00000000
6: 00000004 00000000 00000000 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000000 00000000 00000000 00000000
A: 07300401 0000007F 00000000 00000000
B: 00000000 00000000 00000070 0000000E
C: 00000000 00000000 00000000 00000000
😧 00000000 00000000 00000000 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000121 2C100800
80000002: 65746E49 2952286C 6F655820 2952286E
80000003: 55504320 2D354520 30393632 20347620
80000004: 2E322040 48473036 0000007A 00000000
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 0000302B 00000000 00000000 00000000
terminating...

[sloshburch]

When Splunk dumps the best thing is a support ticket. They have better tools for helping read dumps.

If you find a root cause worth sharing, please do so!

[gjanders]

Please also ensure you are using 7.2.5.1 as 7.2.5.0 has a known bug around FIELDALIAS and crashing...