Getting Data In

New Universal Forwarder read timeout

tinylund
Explorer

We are trying to setup the universal forwarder on a Windows AD server. After configuring the index to receive on port 9997 and installing the UF on the server. The Forwarder does not appear under the Data Inputs/Windows Event Log of Forwarded inputs.

I have verified the firewall is allowing packets on port 9997.
I have verified using tcpdump that packets are being received on port 9997.
I have checked the splunkd.log and found the error indicating TcpInputProc connection from Read Timeout Timed out after 600 seconds.
Documentation indicated sslVersion possible issue - verified the sslVersion on both the inputs.conf of the indexer and the web.conf and outputs.conf of the UF.
Documentation indicated the internal queue on the indexer may be blocked, which causes a timeout after 600 seconds.

How do I find the inrernal queue and troubleshoot if it is blocked?

Thanks in advance for any suggestions.

J

0 Karma

salbro
Path Finder

Not sure what version of Splunk you're using, but when I had indexing issues (recently) I checked the Monitoring Console.

Settings > Monitoring Console (icon) > Indexing > Performance > Indexing Performance: Instance

That view should show the percentage of your queues and where your bottleneck is occurring (if you have one). I'm using Splunk App for Windows, Splunk App for Windows Infrastructure (and the rest of the supporting add-ons), which creates some custom indexes for Windows logs -- not sure if that is something you're interested in or might be helpful.

0 Karma

ddrillic
Ultra Champion

This one can help - I can't find my data!

0 Karma

tinylund
Explorer

resolved the issue by removing the receiving port > restarting the splunk instance - manually adding the port using the CLI splunk enable listen command > restarting the splunk instance

0 Karma

tinylund
Explorer

Those searches just confirmed what I had already indicated, the new Windows server is sending/attempting to send/connect to the Indexer. But there is no metadata of connectivity <600, because the only entries in the log files show errors of Read Time out after 600 sec.

Still not sure how to troubleshoot a blocked queue or how to resolve the queue issue.

Thanks,
J

0 Karma
Get Updates on the Splunk Community!

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...

Introducing the 2024 Splunk MVPs!

We are excited to announce the 2024 cohort of the Splunk MVP program. Splunk MVPs are passionate members of ...

Splunk Custom Visualizations App End of Life

The Splunk Custom Visualizations apps End of Life for SimpleXML will reach end of support on Dec 21, 2024, ...