New HF has all blocked queues in minutes?

[IngmarHicoz]

Hello Splunk Community! I have an ec2 instance of Windows Server 2022 with Splunk Enterprise (9.0.4) installed. Within a few minutes of installing, all of the processing queues are 100% blocked and it places all indexers on quarantine. It is currently outputting to 3 different indexers, and the only logs it is supposed to send is internal logs. I am 100% positive the indexers are not the issue. I think the problem is potentially a connection issue to these indexers as I cannot ping these machines. There is no firewall blocking traffic in between them, so thinking it might be an issue with a setting in server 2022 somewhere.

I made sure to install through Admin CMD line, and for testing, this ec2 has all outbound connections open. Does anyone have any ideas or have seen this before? I had this happen on another box but messing with CMD line and different install flags it finally started working but it seems like no matter what flags I use it doesn't work.

[PickleRick]

If you suspect the network issues, verify the connectivity "the usual way".

Go onto your HF machine and try to connect to your indexers to the input port (usually 9997) and see if it works. If it does, check your _internal log on the indexers for any messages regarding the HF's IP. If it does not... well hard to say without knowing your machines and network setup but generally - something mus be blocking traffic.

Oh, and verify that your indexers do listen on the incoming traffic - I hope someone hasn't configured your boxes to listen on loopback only 🙂

[gcusello]

Hi @IngmarHicoz,

are you receiving internal logs on Indexers?

have you network congestion issues?

what are the hardware onfigurations on Indexers?

Then I saw only test or little environment based on Windows, never production or large environments, only on Linux!

Ciao.

Giuseppe