Deployment Architecture

Why is indexer not discovering other instances?

OgoSplunk
Path Finder

Hi,

 

I'm having some architecture deployment issues on an indexer. When I check hosts using (index="_internal"  | stats count by host). I double-checked my outputs.conf on all my instances, I also checked the inputs.conf on indexer2 both are set to 9997 and I believe I have the right local IP running on all my instances but I still don't have any other host pop-up on Indexer2 except Indexer2. Any assistance would be highly appreciated. Side note I'm using AWS to practice setting up my own Architecture for learning purposes any help will be appreciated.

 

Regards,

Labels (3)
0 Karma
1 Solution

OgoSplunk
Path Finder

No problem long story short the fix to the issue was making sure all my instances where on the same AWS VPC. I'll provide karma.

View solution in original post

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @OgoSplunk,

it seems that your Forwarders don't reach the Indexers:

  • did you open the firewall routes between Forwarders and Indexers?
  • did you disabled local firewalls on the Indexers?
  • Are you using an intermediate Forwarder to concentrate traffic from Forwarders?

In general, you should test the connection from the forwarders using 

telnet ip_indexer 9997

Ciao.

Giuseppe

0 Karma

OgoSplunk
Path Finder

  Hi @gcusello ,

 

Currently, I'm working on the indexer discovering the following: Search Head, Deployment Server, Heavy Forwarder, and Monitoring Console. All local firewalls are disabled I've verified using  (ufw status). I've attached my instance page on AWS to see if maybe you can spot out an issue on the AWS side. 

OgoSplunk_0-1672764979050.png

 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @OgoSplunk,

I suppose that you enabled forwarding to Indexers on all the other Splunk servers.

Then check the connection between servers using telnet .

Did you used IP or name?

Ciao.

Giuseppe

0 Karma

OgoSplunk
Path Finder

@gcusello  Yep correct all systems are forwarding to the indexer and I checked telnet and it stated the following:

splunk@SearchHead:/opt$ telnet 172.31.3.200 9997
Trying 172.31.3.200...
telnet: Unable to connect to remote host: Connection timed out

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @OgoSplunk,

as I said there's something that blocks your connections:

there are thre choicise:

  • the sender,
  • the intermediate network,
  • the receiver.

Abut the first, you have to check local firewall and Splunk receiving, about this, check if the port is the same of sender and if there's SSL enabled in receiving.

Anout netwok, you can check using ntbstats if the packets runs or not.

About the sender, check if the destinations (indexers) addresses and the port are correct; then check if SSL is enabled on Receiver, at least try using Indexers IP address instead hostname.

Ciao.

Giuseppe

OgoSplunk
Path Finder

The error was on the AWS side I have no problem when setting it up on VMWorkstation. The instructor from my lecture didn't include all the additional info needed to set it up Splunk on AWS. I ended up finding Splunk documentation within AWS on how to set up a distributed environment on AWS and it long story short I'm just going to stick with VMWorkstation to save some time. Thanks for your help you're instructions though I'll give you karma for assistance 

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @OgoSplunk,

good for you, see next time!

Please accept one answer for the other people of Community

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

OgoSplunk
Path Finder

No problem long story short the fix to the issue was making sure all my instances where on the same AWS VPC. I'll provide karma.

0 Karma
Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...