topic Re: Why does splunkd.exe have a spike in connections? in Installation

Why does splunkd.exe have a spike in connections?

chanthongphiob — Thu, 17 May 2018 15:21:30 GMT

Why does the "Application Name: \device\harddiskvolume2\program files\splunkuniversalforwarder\bin\splunkd.exe" make excessive connections to the machine? I have run into this issue of it maxing out my license, but would like to know the root cause of excessive connections.

Re: Why does splunkd.exe have a spike in connections?

solarboyz1 — Thu, 17 May 2018 19:17:17 GMT

What do you mean by "make excessive connections to the machine?".

What is excessive? Millions, hundreds, ten?
What type of connection? ldap, wmi, https, etc..?
What machine? Is the universal forwarder running on the machine or connecting to it?

"I have run into this issue of it maxing out my license"

What license is it maxing out

Re: Why does splunkd.exe have a spike in connections?

chanthongphiob — Thu, 17 May 2018 19:33:48 GMT

This is all due to Event ID 5156: The Windows Filtering Platform has allowed a connection. By count of the logs, the number increased. Excessiveness varies according to machines. Machines are Windows boxes. Some machines were running approximately 1 million events everyday but jump to 25 million. Another machine averaged 5000 events per day and increased to 30 million per day.

The event produced includes information with the application name that is executing. The application is "Application Name: \device\harddiskvolume2\program files\splunkuniversalforwarder\bin\splunkd.exe".

As to the type of connection, I am uncertain. The log doesn't specify.

The license is my daily volume license.

I have seen another thread where there was a similar issue but the root cause to why the connections were happening was not given.

Re: Why does splunkd.exe have a spike in connections?

solarboyz1 — Thu, 17 May 2018 19:55:04 GMT

Event ID 5156, documents each time WFP allows a program to connect to another process (on the same or a remote computer) on a TCP or UDP port.

The Splunk UF (\splunkuniversalforwarder\bin\splunkd.exe) connects to the configured Splunk indexers, to send events.

It sounds like you may be running into a loop that is exponentially increasing your logging to Splunk.

An event occurs
Splunk UF connects to indexer to send event
WFP logs 5156 for the connection to the indexers
Splunk UF Connects to indexer to send WFP 5156 event
WFP logs 5156 for the connection to the indexers
Splunk UF Connects to indexer to send WFP 5156 event
etc....

There should be an inputs.conf stanza that defines the monitoring of the windows eventlog that is picking up these events.
You can find that with the following command:

c:\Program Files\SplunkUniversalForwarder\bin>splunk btool --debug inputs list WinEventLog://Security

You will can then blacklist the 5156 events, which will stop them from being ingested by splunk:

inputs.conf:

[WinEventLog://Security]
blacklist1 = EventCode="5156"