Getting Data In

High memory usage by Forwarders due to Indexer unavailability

yZinou
Engager

Hello,

We had a power outage after which our main Splunk instance (which serves as a Search Head and an Indexer) went offline, our Universal Forwarders installed on 2 Windows DC had both a huge memory usage after this causing one of these 2 hosts to crash.

I think the inability to forward events properly caused queues to fill, but checking output.conf file shows an existing reduction setting of their size limit : "maxQueueSize = 100KB" (default is 500KB).

My questions are :

1. What other steps can I take to prevent this ? (I prefer losing logs than having a critical host going offline).
2. Can I set a global memory usage limit for the Forwarders (like 2GB) ?
3. Could this be related to the Forwarders version (7.2.3) ?

I thank you in advance for your support.

Regards.

Labels (1)
Tags (1)
0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

A Four-Part Event Series: State of Observability - Where AI Meets Performance

As AI agents take on more complex tasks, teams need greater visibility into how they behave, what they cost, ...

Persistent Queue at TcpOut — One of Splunk's Most Practical Features

Splunk introduced persistent queueing at the tcpout layer as one of the most practical resilience features in ...

Build resilience for the agentic era: Join us at Splunk Go San Jose

AI is changing how organizations build, operate, and secure digital services. As teams adopt AI agents and ...