Deployment Architecture

Search Head not indexing _internal and summarization errors

mmoermans
Path Finder

Our Splunk Search Head is no longer indexing _internal logs (splunkd.log etc), the searches still run but are really slow. We see the following errors:

08-29-2018 11:08:56.126 +0200 WARN  AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item="").  This may be a bug.
08-29-2018 11:08:58.109 +0200 WARN  AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item="").  This may be a bug.
08-29-2018 11:08:59.044 +0200 WARN  AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item="").  This may be a bug.
08-29-2018 11:09:00.047 +0200 WARN  TcpOutputProc - Forwarding to indexer group cluster blocked for 5560 seconds.

Does anyone know what might be causing this?

Tags (1)
0 Karma

sudosplunk
Motivator

Have a look at splunkd.log on indexers. Are you seeing this issue for only this host (SH) or also for other hosts?

0 Karma

adonio
Ultra Champion

how full is the disk of the search head?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

Hi @mmoermans,

Below log clearly indicate that your search head is not able to send data to Indexer.

08-29-2018 11:09:00.047 +0200 WARN  TcpOutputProc - Forwarding to indexer group cluster blocked for 5560 seconds.

There are multiple reasons for this, for example: firewall block between search head and indexer, Various queue on Indexers are full (Due to low IOPS or higher load on indexer for data processing)

I'll suggest to start with telnet command to check whether connectivity between Search Head and Indexer. For indexers queue status, you can use Monitoring Console if you have configured in your environments.

0 Karma

mstjohn_splunk
Splunk Employee
Splunk Employee

Keep us posted on this @mmoermans, as I'm sure others have a similar question . Good luck and thanks for posting!

0 Karma

mmoermans
Path Finder

Indexer connection is fine:
TcpOutputProc - Connected to idx=ip:port using ACK.

The queue's are empty and our 12 indexers are below 20% CPU usage.
I/O's seem fine as well, is there something else that might be causing this?

0 Karma

harsmarvania57
SplunkTrust
SplunkTrust

In that case I'll suggest to open a support case if you have paid for support.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...