Deployment Architecture

Why am I unable to get index discovery working with ID?

brent_weaver
Builder

I am using index discovery in 2 of my 3 Splunk env's and I have one that will simply not work with ID. I get the following error:

02-03-2018 13:14:23.597 +0000 ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=group1, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK

I have NO IDEA why this is happening, the same automation builds this env as the others.

This is my client config:

[indexer_discovery:idx]
pass4SymmKey = MYSECRET
master_uri = https://myhost:8089

[tcpout:group1]
indexerDiscovery = idx

[tcpout]
defaultGroup = group1

Here is the master index server config:

[indexer_discovery]
pass4SymmKey = MYSECRETKEY
polling_rate = 10
indexerWeightByDiskCapacity = 0

Of course, the keys are hashed. I really hope that someone can figure this one out! I have no clue as it seems like Splunk cannot parse the file or params are missing. I can connect to the master index server from the peers on 8089 without a problem!

Tags (2)
1 Solution

lguinn2
Legend

I think this error message comes up when one of the indexers does not have a receiving port set. In order for indexer discovery to work, all of the indexers must have a receiving port set - the master node collects this information and then supplies it to the forwarder.

It looks like the forwarder connected to the master node correctly (your config files look fine) - but the master couldn't supply the requested information. The "hostport":"?" part of the message is what makes me believe that this is the problem.

View solution in original post

brent_weaver
Builder

Good morning, I did have one indexer that did not have the port listening and I had added the listener service (9997) and I still get the same message?!?!?

0 Karma

lguinn2
Legend

You might want to restart the master node.

0 Karma

lguinn2
Legend

I think this error message comes up when one of the indexers does not have a receiving port set. In order for indexer discovery to work, all of the indexers must have a receiving port set - the master node collects this information and then supplies it to the forwarder.

It looks like the forwarder connected to the master node correctly (your config files look fine) - but the master couldn't supply the requested information. The "hostport":"?" part of the message is what makes me believe that this is the problem.

brent_weaver
Builder

You my friend are an inspiration to people everywhere 🙂
THANK YOU for taking your valuable time to help me out, you were spot on!!!

0 Karma

lguinn2
Legend

(blush) You are welcome

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...