Getting Data In

High memory usage by Forwarders due to Indexer unavailability

yZinou
Engager

Hello,

We had a power outage after which our main Splunk instance (which serves as a Search Head and an Indexer) went offline, our Universal Forwarders installed on 2 Windows DC had both a huge memory usage after this causing one of these 2 hosts to crash.

I think the inability to forward events properly caused queues to fill, but checking output.conf file shows an existing reduction setting of their size limit : "maxQueueSize = 100KB" (default is 500KB).

My questions are :

1. What other steps can I take to prevent this ? (I prefer losing logs than having a critical host going offline).
2. Can I set a global memory usage limit for the Forwarders (like 2GB) ?
3. Could this be related to the Forwarders version (7.2.3) ?

I thank you in advance for your support.

Regards.

Labels (1)
Tags (1)
0 Karma
Get Updates on the Splunk Community!

This Week's Community Digest - Splunk Community Happenings [9.26.22]

Get the latest news and updates from the Splunk Community here! Upcoming User Group Events! 👏 Check ...

BSides Splunk 2022 - The Call for Papers is now Open!

TLDR; Main Site: https://bsidessplunk.com CFP Site: https://bsidessplunk.com/cfp CFP Opens: December 15th, ...

Sending Metrics to Splunk Enterprise With the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...