Splunk caused a kernel panic! What do I do??

[the_wolverine]

I tried installing Splunk today and it appears to have caused a kernel panic. See the kernel panic below:

Pid: 23986, comm: splunkd Not tainted 2.6.9-78.ELsmp
RIP: 0010: [] (next_thread+12)
RAX: ...
RDX: ...



Process splunkd (pid: 23986, threadinfo 00000102e1e54000, task 0000010193d027f0)
Stack: ...

Call Trace:{sys_times+103} {sys_futex+203}
       {do_gettimeofday+77} {system_call+126}
}

Code: ...
RIP 
  Kernel panic - not syncing: Oops

Now what?

[the_wolverine · Accepted solution]

It is unlikely that Splunk is at fault for a kernel panic. Even though a kernel panic may be triggered by a Splunk installation and/or the error points to splunkd, the kernel panic is an indication of a bug in the kernel, not a bug in the installation software.

You can take the information from the error to investigate. For example, from the above, I performed the following search using Google.com:

kernel panic 2.6.9-788.Elsmp

The first url returned is the following RedHat article regarding a sys_times bug in the kernel

"Bug 456993 - kernel panic with kernel version 2.6.9-78.ELsmp" https://bugzilla.redhat.com/show_bug.cgi?id=456993

[ppathik]

I also encountered similar issue with splunk 6.1.3 upgrade, 2.6.18-308.el5

[rroberts]

I had a similar issue today on: Linux ip-10-166-215-67 2.6.21.7-2.ec2.v1.2.fc8xen #1 SMP Fri Nov 20 17:48:28 EST 2009 x86_64 x86_64 x86_64 GNU/Linux with splunkforwarder-5.0.2-149561-Linux-x86_64.tgz.

$SPLUNK_HOME/bin/splunk start

Checking prerequisites...

Checking mgmt port [8089]: open

Message from syslogd@domU-12-31-38-04-18-9C at May 28 19:59:08 ...
kernel:[17884.833628] alignment check: 0000 [#7] SMP

/var/log/dmesg shows... "splunk tainted"