I have a Windows host (192.168.2.2) which has a universal forwarder installed and is setup to talk to my single instance Splunk.
I have added the windows app with only 2 perfmon counters being monitored.
The Windows host run hyper-V which runs the Splunk instance.
As you can see below I have normal connections from other hosts, but for some reason the Windows host has established multiple tcp connections to Splunk. The list of connections keep growing, so I have to stop Splunkd service on the windows host.
netstat -aon | grep 9997
tcp 0 0 0.0.0.0:9997 0.0.0.0:* LISTEN off (0.00/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.2:54228 ESTABLISHED keepalive (6993.74/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.2:54234 ESTABLISHED keepalive (7073.10/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.2:54241 ESTABLISHED keepalive (7132.44/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.2:54216 ESTABLISHED keepalive (6921.62/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.2:54217 ESTABLISHED keepalive (6940.68/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.4:34608 ESTABLISHED keepalive (4530.40/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.102:52379 ESTABLISHED keepalive (4516.28/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.2:54229 ESTABLISHED keepalive (7015.83/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.1:53925 ESTABLISHED keepalive (4518.96/0/0)
tcp 0 0 192.168.2.3:9997 192.168.2.2:54251 ESTABLISHED keepalive (7191.76/0/0)
I have tried restarting Splunkforwarder service, uninstalling and reinstalling Splunk forwarder.
The connections finish but once I start or reinstall the forwarder, the TCP connections start again.
There is nothing special in my etc/system/local/outputs.conf
defaultGroup = default-autolb-group
server = serlin001:9997
Splunk Free Trial 6.5.2
Splunk Universal Forwarder 6.5.2
Not sure what caused this issue but I suspect it was related to the OS. Previously I had restarted Splunkd which did not fix the issue. Later after lodging this case I restarted the operating system which resolved the issue.
Sorry but I do not have a concrete answer. Some troubleshooting steps would be.
Is there anything interesting in the log files on either the UF or the Splunk server? If they're set up with the defaults, check /opt/splunk/var/log/splunk/splunkd.log on the server (Asuming *nix since you used grep 🙂 ) and C:\Program Files\Splunkuniversalforwarder\var\log\splunk\splunkd.log. For the latter, try from an Administrator command prompt
cd \program files\splunkuniversalforwarder\var\log\splunk\ findstr /i tcpout splunkd.log
Though just "type splunkd.log" will likely show you what you need (just scroll back a few pages from the end when it's done). Or even "type splunkd.log | clip" then you should be able to paste that into notepad and look.
I'm not positive what you should see there, but with any luck you'll find errors, warnings, or other text that seems related.