All Apps and Add-ons

Why is the universal forwarder splunkd service using too much RAM on one of 4 domain controllers?

Communicator

We have deployed the TA-DomainController-NT6 add-on from Windows Infrastructure App to 4 of our domain controllers.
In one of the domain controllers the powershell.exe process took more than 6 GB from 8 GB installed. We tried to restart the service but it will go up very fast approximately 1-2 GB per minute. In the other machines we don't face the issue. We have stopped the splunk service due to production impact. Why is it using so much resources ?
Is there anyway to control the amount of RAM used by spkunkd service ?

Another thing to notice is that this server produces a huge amount of logs 12 GB respect to the other servers 1-2 GB

0 Karma

Builder

Does the domain controller in question have a unique role among others? -Check properties of domain in ADUC and determine if the host has any specialized operations master roles.

Are you spreading load among domain controllers to highest degree possible through use of AD Sites and Services and routing of Microsoft client traffic via site-link cost?

Are any 3rd party applications routing directly to the domain controller in question? Can you spread the load of such application traffic across available domain controllers via load balancing or DNS round robin techniques?

0 Karma

Splunk Employee
Splunk Employee

I think it's fair to say that the goal of the app was not to have a 6-8GB powershell process. I think you should determine whether this powershell process is associated with the app (what are its args etc) and work with support to further define the problem.

0 Karma

Legend

"this server produces a huge amount of logs" - is that compared to the other domain controllers or just to another server?

0 Karma

Communicator

Yes compared to other dc-s it generates too much traffic. Can it be that since the server is producing so much load to not overload the network, the data are kept in RAM ? Resulting in Ram overload ?

0 Karma

Communicator

Also if i search the splunkd.log file i can find a lot of errors like this:

ERROR ExecProcessor - message from ""C:\Program Files\SplunkUniversalForwarder\bin\splunk-admon.exe"" splunk-admon - AdQuery::ProcessMessage: Cannot get uSNChanged from message.

Nearly 90% of the entries inside the splunkd.log file. How can we fix this error ?? Can this be the reason ?

0 Karma

Legend

Also - have you looked at the data? Is one DC busier or in trouble?

0 Karma