I am trying to set up a Splunk universal fowarder on a VyOS router going to a Splunk Enterprise instance I have on a Windows 2008 box. The Splunk instance is also connected to a domain that it uses for LDAP authentication for Splunk users and also assigns an IP to the Splunk box via DHCP.
I am trying to get Splunk forwarder traffic over port 8999, which I set in the forwarding and receiving UI of the Splunk. The forwarder shows up in the UI and knows the IP of the router and everything looks fine. However, no traffic is actually getting sent over. I do a netsat -a on the windows Splunk box and it says that there is an established TCP connection on 8089 (management port), but 3 established TCP connections on 8999 (forwarder port) from the router.
The weird part of this is that I had set up a Splunk instance and a VyOS instance, but without having some of the same networking set up (no DHCP, no VLANs on the router) and everything connected and forwarded fine. We had just set up to forward the splunkd.log from the universal forwarder on the router to the main index of the Splunk for proof of life and it was being logged without hindrance.
It doesn't seem to be a firewall, but I guess it very well could be. We are able to ping and do nc -z -v lookups of the IP port for management and receiving and they say they are open. On the working set up there is also only 1 established TCP connection when you netstat -a on the windows Splunk box instead of 3 on the non-working one. Netstat shows established TCP connections on both the router and the Splunk with the same source:destination ports.
By everything we look at it seems like it should be/is connected but we keep getting timeouts in the logs and nothing is getting over. I can provide more info if needed.
We are trying to send data from splunkd.log of the UF which we are monitoring via inputs.conf just as a proof of life. We are trying to send it to the main index not _internal, which i think it defaults to main anyway. We have not seen anything from the UF on the _internal index.
Hey @sdulany, This question is similar and has a few useful tips. If that doesn't do the trick, hopefully one of our community experts will have more ideas. https://answers.splunk.com/answers/417413/why-is-splunk-not-receiving-on-splunktcp-9997.html