All of our Domain Controllers are VMs with limited resources. We have the UF running on them, catching Windows Event Log Security, and admon. The forwarder uses 2.2 GB of memory, and the administrator is flipping out when he sees memory usage warnings from OpenView. Restarting causes memory usage to plunge, but it eventually grows back to this level. I realize that the forwarder is maintaining a cache in case it can't transfer the data up to the indexer. Is there any way to cap this? I have heard that Splunk holds the memory "loosely" and will release it if it is required, but I've got no evidence of that to support my argument.
I got the bad news:
There isn't a way to limit the amount of memory being used by a forwarder directly. What you can do is to turn off inputs and/or descrease the threshold of inputs that run on an interval, like scripted inputs such as perfmon/wmi/etc.
However, Splunk recommends that we use 5.0.3 of the forwarder:
If you want to move to 5.0.3, that is fine too, it contains a number of fixes for splunk-admon.exe related to memory.
So I'm not happy about the memory use, but I'd rather have the log data.
So you get the memory usage problem on both DCs over LAN and WAN, right ?
When you perform a search targeting one of this DC, do you see a big time gap between the time you run the search and the timestamp of the last search results ?
If yes, I'd say either the forwarder can't keep up with the events rate through the WAN, or the indexer can't keep up generally with the amount of incoming events.
One of those bottlenecks could explain the huge memory usage ?
I saw a note about that, but it seemed that would only reduce memory usage by a few MB. I need to reduce by a GB.
Inputs are WinEventLog:Security and the ad monitor.
Indexer acknowledgement is a new one on me!
I'm sure that everything is at a default value