Resolving missing logs when my connections and configs seems fine.

[dantimola]

Hi All,

We have missing logs from a DHCP server that has a splunk forwarder installed, the network connectivity is fine, configs are also fine, firewall is also allowed, however, when I checked the splunkd.log, I still saw -0500 WARN TcpOutputProc - Raw connection to ip=xx.xx.xx.xx:9997 timed out and 0500 WARN TcpOutputProc - Cooked connection to ip=xx.xx.xx.xx:9997 timed out
We performed initial troubleshooting and the results are as follows:

1. The connectivity from the two servers are established both in our DS and HF and yet we still haven't got any logs

2. The log file is right and currently active during this time

3. Configs on inputs and outputs are also proper

Thanks in advance

[gcusello]

Hi dantimola,
just some stupid answers:

• did you checked if you're receiving Splunk logs (index=_internal)?
• did you checked if the local firewall is open for 9997 and 8089 ports?
• did you checked firewall rules for ports 9997 and 8089 with telnet?
• are you using SSL?

at a first sight, it seems that your forwarders cannot reach to be connected with Indexers

Bye.
Giuseppe

[3no]

Assuming your Splunk Architecture in based on Linux.

Try with tcpdump on your indexer to see if the logs are arriving, if yes then check that your sending the events in the proper index.

Tcpdump command :

tcpdump -ni [name_of_interface] host [ip_of_your_forwarder] 

To find the name of your interface just make a ifconfig.

3no.

[3no]

And also check, that your firewall or loadbalancer as not a limit in the TCP timeout session, it happens sometimes after a certain amount of time the firewall close the connection.