All Apps and Add-ons

Why is the universal forwarder splunkd service using too much RAM on one of 4 domain controllers?

arber
Communicator

We have deployed the TA-DomainController-NT6 add-on from Windows Infrastructure App to 4 of our domain controllers.
In one of the domain controllers the powershell.exe process took more than 6 GB from 8 GB installed. We tried to restart the service but it will go up very fast approximately 1-2 GB per minute. In the other machines we don't face the issue. We have stopped the splunk service due to production impact. Why is it using so much resources ?
Is there anyway to control the amount of RAM used by spkunkd service ?

Another thing to notice is that this server produces a huge amount of logs 12 GB respect to the other servers 1-2 GB

0 Karma

dstaulcu
Builder

Does the domain controller in question have a unique role among others? -Check properties of domain in ADUC and determine if the host has any specialized operations master roles.

Are you spreading load among domain controllers to highest degree possible through use of AD Sites and Services and routing of Microsoft client traffic via site-link cost?

Are any 3rd party applications routing directly to the domain controller in question? Can you spread the load of such application traffic across available domain controllers via load balancing or DNS round robin techniques?

0 Karma

jrodman
Splunk Employee
Splunk Employee

I think it's fair to say that the goal of the app was not to have a 6-8GB powershell process. I think you should determine whether this powershell process is associated with the app (what are its args etc) and work with support to further define the problem.

0 Karma

lguinn2
Legend

"this server produces a huge amount of logs" - is that compared to the other domain controllers or just to another server?

0 Karma

arber
Communicator

Yes compared to other dc-s it generates too much traffic. Can it be that since the server is producing so much load to not overload the network, the data are kept in RAM ? Resulting in Ram overload ?

0 Karma

arber
Communicator

Also if i search the splunkd.log file i can find a lot of errors like this:

ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-admon.exe"" splunk-admon - AdQuery::ProcessMessage: Cannot get uSNChanged from message.

Nearly 90% of the entries inside the splunkd.log file. How can we fix this error ?? Can this be the reason ?

0 Karma

lguinn2
Legend

Also - have you looked at the data? Is one DC busier or in trouble?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...