topic typing queue always full with indexer crash in Getting Data In

typing queue always full with indexer crash

surekhasplunk — Mon, 23 Mar 2020 11:18:56 GMT

Once my indexer crashed with below error:
kernel: splunkd[] general protection ip:xyz error:0 in splunkd[]

And after restarting the indexer my Parsing, Merging , Typing queues are always full with going to the indexing queue.

How to resolve this issue ?

Re: typing queue always full with indexer crash

anmolpatel — Mon, 23 Mar 2020 12:16:59 GMT

@surekhasplunk Potential things to check:
- the transforms that are applied to the sourcetypes, there could be issues with that if you've created custom transforms or routing of the data
- SED extraction in props.conf

Re: typing queue always full with indexer crash

PavelP — Mon, 23 Mar 2020 13:13:24 GMT

General protection error is a kernel message, I suspect it is not a configuration issue. You have to open a support ticket. To confirm it:

check logs for similar errors with "grep kernel | grep error" - do you see other errors? are all of them related to splunk or there are other processes?
check for crash* logs in $SPLUNK_HOME/var/log/splunk
check /var/log/messages or journal for OOM or similar OS errors

Re: typing queue always full with indexer crash

surekhasplunk — Wed, 30 Sep 2020 04:45:23 GMT

Hi @PaveIP

I raised case with support but they are going in completely different direction all together instead of finding the root cause of this particular problem they are like change the whole setup upgrade or move conf files etc.
We can do that but not now. Not at that time when the crash happened but at a later point.

1) And i did check log for kernel or error and found something like below
16:00:53 SPKLX kernel: device bond0 left promiscuous mode
16:00:53 SPKLX kernel: device em1 left promiscuous mode
16:00:53 SPKLX kernel: device bond0 entered promiscuous mode
16:00:53 SPKLX kernel: device em1 entered promiscuous mode

2) i checked for crash.log and they have :
Received fatal signal 11 (Segmentation fault).
Cause:
Unknown signal origin (si_code=128, si_addr=[0x0000000000000000]).
Crashing thread: CallbackRunnerThread
Registers:
RIP: [0x00007F719784F505]

3)i cant see any OOM or OS errors.

Re: typing queue always full with indexer crash

PavelP — Tue, 24 Mar 2020 07:49:48 GMT

thank you @surekhasplunk for an update

1) information messages related to tcpdump
2) if all crashlogs have "CallbackRunnerThread" as a cause it could be splunk fault and an update can help
3) -

good luck with troubleshooting!