Hi Increasing autoLBFrequency, maxQueueSize, aggQueueSize, or outputQueueSize in outputs.conf on your heavy forwarders may help temporarily reduce "blocked=true" messages, but these settings do not ...
See more...
Hi Increasing autoLBFrequency, maxQueueSize, aggQueueSize, or outputQueueSize in outputs.conf on your heavy forwarders may help temporarily reduce "blocked=true" messages, but these settings do not address the root cause: your indexer(s) are overloaded and unable to keep up with incoming data. The following will tell you which queues are blocking on which servers: index=_internal source=*metrics.log blocked=true
| stats count by host, group, name "blocked=true" in metrics.log means the forwarder cannot send data to the indexer because the indexer is not accepting it fast enough (usually due to CPU, disk, or queue saturation). Increasing forwarder queue sizes only buffers more data; it does not fix indexer bottlenecks. The indexer with 99–100% CPU is a clear bottleneck. Upgrading its CPU may help, but if the load is not balanced across all indexers, you may need to investigate why (e.g., uneven load balancing, hot buckets, or misconfiguration). Lowering autoLBFrequency (e.g., from 10 to 5) can help distribute load more evenly, but will not solve indexer resource exhaustion. Do not rely solely on queue size increases; this can delay but not prevent data loss if indexers remain overloaded. Investigate why one indexer is overloaded (check for hot buckets, network issues, or misconfigured load balancing). Understanding *why* the single indexer is blocking is probably the important thing here - it could be a number of things but likely to be either resource issue (e.g. faulty disk) or one of your syslog feeds failing to balance to another indexer. Is it always the same indexer that runs hot? Or does it change? Did this answer help you? If so, please consider: Adding karma to show it was useful Marking it as the solution if it resolved your issue Commenting if you need any clarification Your feedback encourages the volunteers in this community to continue contributing