Getting Data In

typing queue always full with indexer crash

surekhasplunk
Communicator

Hi

Once my indexer crashed with below error:
kernel: splunkd[] general protection ip:xyz error:0 in splunkd[]

And after restarting the indexer my Parsing, Merging , Typing queues are always full with going to the indexing queue.

How to resolve this issue ?

Tags (2)
0 Karma
1 Solution

PavelP
Motivator

General protection error is a kernel message, I suspect it is not a configuration issue. You have to open a support ticket. To confirm it:

  • check logs for similar errors with "grep kernel | grep error" - do you see other errors? are all of them related to splunk or there are other processes?
  • check for crash* logs in $SPLUNK_HOME/var/log/splunk
  • check /var/log/messages or journal for OOM or similar OS errors

View solution in original post

0 Karma

PavelP
Motivator

General protection error is a kernel message, I suspect it is not a configuration issue. You have to open a support ticket. To confirm it:

  • check logs for similar errors with "grep kernel | grep error" - do you see other errors? are all of them related to splunk or there are other processes?
  • check for crash* logs in $SPLUNK_HOME/var/log/splunk
  • check /var/log/messages or journal for OOM or similar OS errors
0 Karma

surekhasplunk
Communicator

Hi @PaveIP

I raised case with support but they are going in completely different direction all together instead of finding the root cause of this particular problem they are like change the whole setup upgrade or move conf files etc.
We can do that but not now. Not at that time when the crash happened but at a later point.

1) And i did check log for kernel or error and found something like below
16:00:53 SPKLX kernel: device bond0 left promiscuous mode
16:00:53 SPKLX kernel: device em1 left promiscuous mode
16:00:53 SPKLX kernel: device bond0 entered promiscuous mode
16:00:53 SPKLX kernel: device em1 entered promiscuous mode

2) i checked for crash.log and they have :
Received fatal signal 11 (Segmentation fault).
Cause:
Unknown signal origin (si_code=128, si_addr=[0x0000000000000000]).
Crashing thread: CallbackRunnerThread
Registers:
RIP: [0x00007F719784F505]

3)i cant see any OOM or OS errors.

0 Karma

PavelP
Motivator

thank you @surekhasplunk for an update

1) information messages related to tcpdump
2) if all crashlogs have "CallbackRunnerThread" as a cause it could be splunk fault and an update can help
3) -

good luck with troubleshooting!

0 Karma

anmolpatel
Builder

@surekhasplunk Potential things to check:
- the transforms that are applied to the sourcetypes, there could be issues with that if you've created custom transforms or routing of the data
- SED extraction in props.conf

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...