Why is indexer not discovering other instances?

[OgoSplunk]

Hi,

I'm having some architecture deployment issues on an indexer. When I check hosts using (index="_internal"  | stats count by host). I double-checked my outputs.conf on all my instances, I also checked the inputs.conf on indexer2 both are set to 9997 and I believe I have the right local IP running on all my instances but I still don't have any other host pop-up on Indexer2 except Indexer2. Any assistance would be highly appreciated. Side note I'm using AWS to practice setting up my own Architecture for learning purposes any help will be appreciated.

Regards,

[gcusello]

Hi @OgoSplunk,

it seems that your Forwarders don't reach the Indexers:

• did you open the firewall routes between Forwarders and Indexers?
• did you disabled local firewalls on the Indexers?
• Are you using an intermediate Forwarder to concentrate traffic from Forwarders?

In general, you should test the connection from the forwarders using 

telnet ip_indexer 9997

Ciao.

Giuseppe

[OgoSplunk]

  Hi @gcusello ,

Currently, I'm working on the indexer discovering the following: Search Head, Deployment Server, Heavy Forwarder, and Monitoring Console. All local firewalls are disabled I've verified using  (ufw status). I've attached my instance page on AWS to see if maybe you can spot out an issue on the AWS side. 

[gcusello]

Hi @OgoSplunk,

I suppose that you enabled forwarding to Indexers on all the other Splunk servers.

Then check the connection between servers using telnet .

Did you used IP or name?

Ciao.

Giuseppe

[OgoSplunk]

@gcusello  Yep correct all systems are forwarding to the indexer and I checked telnet and it stated the following:

splunk@SearchHead:/opt$ telnet 172.31.3.200 9997
Trying 172.31.3.200...
telnet: Unable to connect to remote host: Connection timed out

[gcusello]

Hi @OgoSplunk,

as I said there's something that blocks your connections:

there are thre choicise:

• the sender,
• the intermediate network,
• the receiver.

Abut the first, you have to check local firewall and Splunk receiving, about this, check if the port is the same of sender and if there's SSL enabled in receiving.

Anout netwok, you can check using ntbstats if the packets runs or not.

About the sender, check if the destinations (indexers) addresses and the port are correct; then check if SSL is enabled on Receiver, at least try using Indexers IP address instead hostname.

Ciao.

Giuseppe

[OgoSplunk]

The error was on the AWS side I have no problem when setting it up on VMWorkstation. The instructor from my lecture didn't include all the additional info needed to set it up Splunk on AWS. I ended up finding Splunk documentation within AWS on how to set up a distributed environment on AWS and it long story short I'm just going to stick with VMWorkstation to save some time. Thanks for your help you're instructions though I'll give you karma for assistance 

[gcusello]

Hi @OgoSplunk,

good for you, see next time!

Please accept one answer for the other people of Community

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[OgoSplunk · Accepted solution]

No problem long story short the fix to the issue was making sure all my instances where on the same AWS VPC. I'll provide karma.