Search Head not indexing _internal and summarization errors

[mmoermans]

Our Splunk Search Head is no longer indexing _internal logs (splunkd.log etc), the searches still run but are really slow. We see the following errors:

08-29-2018 11:08:56.126 +0200 WARN  AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item="").  This may be a bug.
08-29-2018 11:08:58.109 +0200 WARN  AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item="").  This may be a bug.
08-29-2018 11:08:59.044 +0200 WARN  AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item="").  This may be a bug.
08-29-2018 11:09:00.047 +0200 WARN  TcpOutputProc - Forwarding to indexer group cluster blocked for 5560 seconds.

Does anyone know what might be causing this?

[sudosplunk]

Have a look at splunkd.log on indexers. Are you seeing this issue for only this host (SH) or also for other hosts?

[adonio]

how full is the disk of the search head?

[harsmarvania57]

Hi @mmoermans,

Below log clearly indicate that your search head is not able to send data to Indexer.

08-29-2018 11:09:00.047 +0200 WARN  TcpOutputProc - Forwarding to indexer group cluster blocked for 5560 seconds.

There are multiple reasons for this, for example: firewall block between search head and indexer, Various queue on Indexers are full (Due to low IOPS or higher load on indexer for data processing)

I'll suggest to start with telnet command to check whether connectivity between Search Head and Indexer. For indexers queue status, you can use Monitoring Console if you have configured in your environments.

[mstjohn_splunk]

Keep us posted on this @mmoermans, as I'm sure others have a similar question . Good luck and thanks for posting!

[mmoermans]

Indexer connection is fine:
TcpOutputProc - Connected to idx=ip:port using ACK.

The queue's are empty and our 12 indexers are below 20% CPU usage.
I/O's seem fine as well, is there something else that might be causing this?

[harsmarvania57]

In that case I'll suggest to open a support case if you have paid for support.