Deployment Architecture

Splunk search head suddenly crashed

Communicator

hi - last week our splunk search head instance suddenly crashed. we have the below log from the crash log file generated. Unfortunately, we dont have the knowledge to read the log and understand. Please help!

Received fatal signal 6 (Aborted).
Cause:
Signal sent by PID 15097 running under UID 1030.
Crashing thread: TcpChannelThread
Registers:
RIP: [0x00007F4A91D4B1F7] gsignal + 55 (libc.so.6 + 0x351F7)
RDI: [0x0000000000003AF9]
RSI: [0x0000000000004764]
RBP: [0x00007F4A91E96E68]
RSP: [0x00007F4A3B3FD8F8]
RAX: [0x0000000000000000]
RBX: [0x00007F4A86B66000]
RCX: [0xFFFFFFFFFFFFFFFF]
RDX: [0x0000000000000006]
R8: [0x0000000000000060]
R9: [0xFEFEFEFEFF092D63]
R10: [0x0000000000000008]
R11: [0x0000000000000206]
R12: [0x000055FE4823AC14]
R13: [0x000055FE483D8BC0]
R14: [0x00007F4A78544288]
R15: [0x00007F4A3B3FDC20]
EFL: [0x0000000000000206]
TRAPNO: [0x0000000000000000]
ERR: [0x0000000000000000]
CSGSFS: [0x0000000000000033]
OLDMASK: [0x0000000000000000]

OS: Linux
Arch: x86-64

Backtrace (PIC build):
Linux / prd-usc1-a-splunk-nonessh / 3.10.0-693.5.2.el7.x8664 / #1 SMP Fri Oct 13 10:46:25 EDT 2017 / x8664
glibc version: 2.17
glibc release: stable
Last errno: 24
Threads running: 252
Runtime: 27459.529833s
argv: [splunkd -p 8089 restart]
Regex JIT enabled

x86 CPUID registers:
0: 0000000D 756E6547 6C65746E 49656E69
1: 000206D7 0E200800 9EB82203 1F8BFBFF
2: 76035A01 00F0B2FF 00000000 00CA0000
3: 00000000 00000000 00000000 00000000
4: 00000000 00000000 00000000 00000000
5: 00000000 00000000 00000000 00000000
6: 00000000 00000000 00000000 00000000
7: 00000000 00000000 00000000 00000000
8: 00000000 00000000 00000000 00000000
9: 00000001 00000000 00000000 00000000
A: 07300000 00000000 00000000 00000603
B: 00000000 00000000 00000000 00000000
C: 00000000 00000000 00000000 00000000
😧 00000000 00000000 00000000 00000000
80000000: 80000008 00000000 00000000 00000000
80000001: 00000000 00000000 00000001 2C100800
80000002: 20202020 20202020 20202020 20202020
80000003: 746E4920 52286C65 65582029 52286E6F
80000004: 50432029 20402055 30362E32 007A4847
80000005: 00000000 00000000 00000000 00000000
80000006: 00000000 00000000 01006040 00000000
80000007: 00000000 00000000 00000000 00000100
80000008: 0000302E 00000000 00000000 00000000
terminating...

0 Karma

Influencer

Crashes are definitely something to raise a support case about with a diag and as much as information as you can figure out about how to reproduce such. They'd have the tools to line up the crash with the actual code for your specific version of Splunk that you have installed... Signal 6 (SIGABRT) can be problems in usage of internal libraries / memory management or a number of other things per this Stack overflow thread: https://stackoverflow.com/q/3413166/504685

Well assuming that process id 15097 was the splunk process, and UID 1030 is the user id of the splunk user of course.

A previous splunk answers post with no accepted answer seems to point toward an app install workflow, but unknown what version of Splunk and if your site specific configurations make it more likely or not and if it's a similar issue or not: https://answers.splunk.com/answers/581491/splunk-crashes-when-trying-to-install-an-app-from.html

I would also recommend that you edit down and redact some of the content in your post a bit here, as well as the copy you posted on Stack Overflow: https://stackoverflow.com/q/50441355/504685 session_id values are typically not the type of data you want to post publicly on the internet.

Communicator

thanks so much, i'll take your advise about the diag and opening a support case as we cant really read and understand the log 🙂

0 Karma

Ultra Champion

Yay! Support ftw!

0 Karma

Communicator

unfortunately, I dont have access to upload and create cases .
as per Splunk customer support, i dont have active support contract or entitlement. 😞

0 Karma

Ultra Champion

Are you not a paying customer of support? Or are there other admins at your account that can create the support account for you?

0 Karma

Communicator

It's the client's instance, we are just developing what was asked to us.

0 Karma

SplunkTrust
SplunkTrust

@teddyidc1101 check with Clients whether they have active support entitlement and that issues like this should be handled only by working with the Splunk Support Team.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma

Communicator

yeah, that's what i also thought. thanks!

0 Karma

SplunkTrust
SplunkTrust

@teddyidc1101 All the best! Once it gets fixed, do post the cause and resolution and any other useful details, if you get to know for the interest of others facing similar issue.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

Path Finder

agreed! TMI here!

Open a support case with Splunk.

SplunkTrust
SplunkTrust

@ehollima, thanks for supporting the answer by @acharlieh by giving points. Just a hint, if you like the answer/comment on the community, you can definitely Up Vote the same using Up Arrow icon next to answer or for comments which shows up on hovering.

PS: While Down Voting is also possible it should be restricted to worst case scenarios like an harmful suggestion which may break your system. In order to keep the positive vibe alive on the community, we generally do not down vote and comment/answer against the existing answer to support our thinking/correct approach. Refer to the community guidelines.

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
0 Karma