Installation

New HF has all blocked queues in minutes?

IngmarHicoz
Engager

Hello Splunk Community! I have an ec2 instance of Windows Server 2022 with Splunk Enterprise (9.0.4) installed. Within a few minutes of installing, all of the processing queues are 100% blocked and it places all indexers on quarantine. It is currently outputting to 3 different indexers, and the only logs it is supposed to send is internal logs. I am 100% positive the indexers are not the issue. I think the problem is potentially a connection issue to these indexers as I cannot ping these machines. There is no firewall blocking traffic in between them, so thinking it might be an issue with a setting in server 2022 somewhere.

I made sure to install through Admin CMD line, and for testing, this ec2 has all outbound connections open. Does anyone have any ideas or have seen this before? I had this happen on another box but messing with CMD line and different install flags it finally started working but it seems like no matter what flags I use it doesn't work.

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

If you suspect the network issues, verify the connectivity "the usual way".

Go onto your HF machine and try to connect to your indexers to the input port (usually 9997) and see if it works. If it does, check your _internal log on the indexers for any messages regarding the HF's IP. If it does not... well hard to say without knowing your machines and network setup but generally - something mus be blocking traffic.

Oh, and verify that your indexers do listen on the incoming traffic - I hope someone hasn't configured your boxes to listen on loopback only 🙂

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @IngmarHicoz,

are you receiving internal logs on Indexers?

have you network congestion issues?

what are the hardware onfigurations on Indexers?

Then I saw only test or little environment based on Windows, never production or large environments, only on Linux!

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...