Join the Conversation
- Find Answers
- :
- Splunk Administration
- :
- Deployment Architecture
- :
- Search Head not indexing _internal and summarizati...
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark Topic
- Subscribe to Topic
- Mute Topic
- Printer Friendly Page
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Search Head not indexing _internal and summarization errors
Our Splunk Search Head is no longer indexing _internal logs (splunkd.log etc), the searches still run but are really slow. We see the following errors:
08-29-2018 11:08:56.126 +0200 WARN AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item=""). This may be a bug.
08-29-2018 11:08:58.109 +0200 WARN AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item=""). This may be a bug.
08-29-2018 11:08:59.044 +0200 WARN AdminManager - Handler 'summarization' has not performed any capability checks for this operation (requestedAction=list, customAction="", item=""). This may be a bug.
08-29-2018 11:09:00.047 +0200 WARN TcpOutputProc - Forwarding to indexer group cluster blocked for 5560 seconds.
Does anyone know what might be causing this?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Have a look at splunkd.log on indexers. Are you seeing this issue for only this host (SH) or also for other hosts?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
how full is the disk of the search head?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @mmoermans,
Below log clearly indicate that your search head is not able to send data to Indexer.
08-29-2018 11:09:00.047 +0200 WARN TcpOutputProc - Forwarding to indexer group cluster blocked for 5560 seconds.
There are multiple reasons for this, for example: firewall block between search head and indexer, Various queue on Indexers are full (Due to low IOPS or higher load on indexer for data processing)
I'll suggest to start with telnet command to check whether connectivity between Search Head and Indexer. For indexers queue status, you can use Monitoring Console if you have configured in your environments.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Keep us posted on this @mmoermans, as I'm sure others have a similar question . Good luck and thanks for posting!
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Indexer connection is fine:
TcpOutputProc - Connected to idx=ip:port using ACK.
The queue's are empty and our 12 indexers are below 20% CPU usage.
I/O's seem fine as well, is there something else that might be causing this?
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
In that case I'll suggest to open a support case if you have paid for support.
Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!
Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.
Announcing Modern Navigation: A New Era of Splunk User Experience
Modernize your Splunk Apps – Introducing Python 3.13 in Splunk
Step into “Hunt the Insider: An Splunk ES Premier Mystery” to catch a cybercriminal ...
