typing queue always full with indexer crash

[surekhasplunk]

Hi

Once my indexer crashed with below error:
kernel: splunkd[] general protection ip:xyz error:0 in splunkd[]

And after restarting the indexer my Parsing, Merging , Typing queues are always full with going to the indexing queue.

How to resolve this issue ?

[PavelP · Accepted solution]

General protection error is a kernel message, I suspect it is not a configuration issue. You have to open a support ticket. To confirm it:

• check logs for similar errors with "grep kernel | grep error" - do you see other errors? are all of them related to splunk or there are other processes?
  
• check for crash* logs in $SPLUNK_HOME/var/log/splunk
• check /var/log/messages or journal for OOM or similar OS errors

[surekhasplunk]

Hi @PaveIP

I raised case with support but they are going in completely different direction all together instead of finding the root cause of this particular problem they are like change the whole setup upgrade or move conf files etc.
We can do that but not now. Not at that time when the crash happened but at a later point.

1) And i did check log for kernel or error and found something like below
16:00:53 SPKLX kernel: device bond0 left promiscuous mode
16:00:53 SPKLX kernel: device em1 left promiscuous mode
16:00:53 SPKLX kernel: device bond0 entered promiscuous mode
16:00:53 SPKLX kernel: device em1 entered promiscuous mode

2) i checked for crash.log and they have :
Received fatal signal 11 (Segmentation fault).
Cause:
Unknown signal origin (si_code=128, si_addr=[0x0000000000000000]).
Crashing thread: CallbackRunnerThread
Registers:
RIP: [0x00007F719784F505]

3)i cant see any OOM or OS errors.

[PavelP]

thank you @surekhasplunk for an update

1) information messages related to tcpdump
2) if all crashlogs have "CallbackRunnerThread" as a cause it could be splunk fault and an update can help
3) -

good luck with troubleshooting!

[anmolpatel]

@surekhasplunk Potential things to check:
- the transforms that are applied to the sourcetypes, there could be issues with that if you've created custom transforms or routing of the data
- SED extraction in props.conf