We have missing logs from a DHCP server that has a splunk forwarder installed, the network connectivity is fine, configs are also fine, firewall is also allowed, however, when I checked the splunkd.log, I still saw
-0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out and
0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out
We performed initial troubleshooting and the results are as follows:
The connectivity from the two servers are established both in our DS and HF and yet we still haven't got any logs
The log file is right and currently active during this time
Configs on inputs and outputs are also proper
Thanks in advance
Assuming your Splunk Architecture in based on Linux.
Try with tcpdump on your indexer to see if the logs are arriving, if yes then check that your sending the events in the proper index.
Tcpdump command :
tcpdump -ni [name_of_interface] host [ip_of_your_forwarder]
To find the name of your interface just make a
And also check, that your firewall or loadbalancer as not a limit in the TCP timeout session, it happens sometimes after a certain amount of time the firewall close the connection.
just some stupid answers:
at a first sight, it seems that your forwarders cannot reach to be connected with Indexers