Getting Data In

typing queue always full with indexer crash

surekhasplunk
Communicator

Hi

Once my indexer crashed with below error:
kernel: splunkd[] general protection ip:xyz error:0 in splunkd[]

And after restarting the indexer my Parsing, Merging , Typing queues are always full with going to the indexing queue.

How to resolve this issue ?

Tags (2)
0 Karma
1 Solution

PavelP
Motivator

General protection error is a kernel message, I suspect it is not a configuration issue. You have to open a support ticket. To confirm it:

  • check logs for similar errors with "grep kernel | grep error" - do you see other errors? are all of them related to splunk or there are other processes?
  • check for crash* logs in $SPLUNK_HOME/var/log/splunk
  • check /var/log/messages or journal for OOM or similar OS errors

View solution in original post

0 Karma

PavelP
Motivator

General protection error is a kernel message, I suspect it is not a configuration issue. You have to open a support ticket. To confirm it:

  • check logs for similar errors with "grep kernel | grep error" - do you see other errors? are all of them related to splunk or there are other processes?
  • check for crash* logs in $SPLUNK_HOME/var/log/splunk
  • check /var/log/messages or journal for OOM or similar OS errors
0 Karma

surekhasplunk
Communicator

Hi @PaveIP

I raised case with support but they are going in completely different direction all together instead of finding the root cause of this particular problem they are like change the whole setup upgrade or move conf files etc.
We can do that but not now. Not at that time when the crash happened but at a later point.

1) And i did check log for kernel or error and found something like below
16:00:53 SPKLX kernel: device bond0 left promiscuous mode
16:00:53 SPKLX kernel: device em1 left promiscuous mode
16:00:53 SPKLX kernel: device bond0 entered promiscuous mode
16:00:53 SPKLX kernel: device em1 entered promiscuous mode

2) i checked for crash.log and they have :
Received fatal signal 11 (Segmentation fault).
Cause:
Unknown signal origin (si_code=128, si_addr=[0x0000000000000000]).
Crashing thread: CallbackRunnerThread
Registers:
RIP: [0x00007F719784F505]

3)i cant see any OOM or OS errors.

0 Karma

PavelP
Motivator

thank you @surekhasplunk for an update

1) information messages related to tcpdump
2) if all crashlogs have "CallbackRunnerThread" as a cause it could be splunk fault and an update can help
3) -

good luck with troubleshooting!

0 Karma

anmolpatel
Builder

@surekhasplunk Potential things to check:
- the transforms that are applied to the sourcetypes, there could be issues with that if you've created custom transforms or routing of the data
- SED extraction in props.conf

0 Karma
Get Updates on the Splunk Community!

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...

Splunkbase | Splunk Dashboard Examples App for SimpleXML End of Life

The Splunk Dashboard Examples App for SimpleXML will reach end of support on Dec 19, 2024, after which no new ...

Understanding Generative AI Techniques and Their Application in Cybersecurity

Watch On-Demand Artificial intelligence is the talk of the town nowadays, with industries of all kinds ...