Getting Data In

New Universal Forwarder read timeout

tinylund
Explorer

We are trying to setup the universal forwarder on a Windows AD server. After configuring the index to receive on port 9997 and installing the UF on the server. The Forwarder does not appear under the Data Inputs/Windows Event Log of Forwarded inputs.

I have verified the firewall is allowing packets on port 9997.
I have verified using tcpdump that packets are being received on port 9997.
I have checked the splunkd.log and found the error indicating TcpInputProc connection from Read Timeout Timed out after 600 seconds.
Documentation indicated sslVersion possible issue - verified the sslVersion on both the inputs.conf of the indexer and the web.conf and outputs.conf of the UF.
Documentation indicated the internal queue on the indexer may be blocked, which causes a timeout after 600 seconds.

How do I find the inrernal queue and troubleshoot if it is blocked?

Thanks in advance for any suggestions.

J

0 Karma

salbro
Path Finder

Not sure what version of Splunk you're using, but when I had indexing issues (recently) I checked the Monitoring Console.

Settings > Monitoring Console (icon) > Indexing > Performance > Indexing Performance: Instance

That view should show the percentage of your queues and where your bottleneck is occurring (if you have one). I'm using Splunk App for Windows, Splunk App for Windows Infrastructure (and the rest of the supporting add-ons), which creates some custom indexes for Windows logs -- not sure if that is something you're interested in or might be helpful.

0 Karma

ddrillic
Ultra Champion

This one can help - I can't find my data!

0 Karma

tinylund
Explorer

resolved the issue by removing the receiving port > restarting the splunk instance - manually adding the port using the CLI splunk enable listen command > restarting the splunk instance

0 Karma

tinylund
Explorer

Those searches just confirmed what I had already indicated, the new Windows server is sending/attempting to send/connect to the Indexer. But there is no metadata of connectivity <600, because the only entries in the log files show errors of Read Time out after 600 sec.

Still not sure how to troubleshoot a blocked queue or how to resolve the queue issue.

Thanks,
J

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...