New Universal Forwarder read timeout

[tinylund]

We are trying to setup the universal forwarder on a Windows AD server. After configuring the index to receive on port 9997 and installing the UF on the server. The Forwarder does not appear under the Data Inputs/Windows Event Log of Forwarded inputs.

I have verified the firewall is allowing packets on port 9997.
I have verified using tcpdump that packets are being received on port 9997.
I have checked the splunkd.log and found the error indicating TcpInputProc connection from Read Timeout Timed out after 600 seconds.
Documentation indicated sslVersion possible issue - verified the sslVersion on both the inputs.conf of the indexer and the web.conf and outputs.conf of the UF.
Documentation indicated the internal queue on the indexer may be blocked, which causes a timeout after 600 seconds.

How do I find the inrernal queue and troubleshoot if it is blocked?

Thanks in advance for any suggestions.

J

[salbro]

Not sure what version of Splunk you're using, but when I had indexing issues (recently) I checked the Monitoring Console.

Settings > Monitoring Console (icon) > Indexing > Performance > Indexing Performance: Instance

That view should show the percentage of your queues and where your bottleneck is occurring (if you have one). I'm using Splunk App for Windows, Splunk App for Windows Infrastructure (and the rest of the supporting add-ons), which creates some custom indexes for Windows logs -- not sure if that is something you're interested in or might be helpful.

[ddrillic]

This one can help - I can't find my data!

[tinylund]

resolved the issue by removing the receiving port > restarting the splunk instance - manually adding the port using the CLI splunk enable listen command > restarting the splunk instance

[tinylund]

Those searches just confirmed what I had already indicated, the new Windows server is sending/attempting to send/connect to the Indexer. But there is no metadata of connectivity <600, because the only entries in the log files show errors of Read Time out after 600 sec.

Still not sure how to troubleshoot a blocked queue or how to resolve the queue issue.

Thanks,
J