Deployment Architecture

Why am I unable to get index discovery working with ID?

brent_weaver
Builder

I am using index discovery in 2 of my 3 Splunk env's and I have one that will simply not work with ID. I get the following error:

02-03-2018 13:14:23.597 +0000 ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=group1, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK

I have NO IDEA why this is happening, the same automation builds this env as the others.

This is my client config:

[indexer_discovery:idx]
pass4SymmKey = MYSECRET
master_uri = https://myhost:8089

[tcpout:group1]
indexerDiscovery = idx

[tcpout]
defaultGroup = group1

Here is the master index server config:

[indexer_discovery]
pass4SymmKey = MYSECRETKEY
polling_rate = 10
indexerWeightByDiskCapacity = 0

Of course, the keys are hashed. I really hope that someone can figure this one out! I have no clue as it seems like Splunk cannot parse the file or params are missing. I can connect to the master index server from the peers on 8089 without a problem!

Tags (2)
1 Solution

lguinn2
Legend

I think this error message comes up when one of the indexers does not have a receiving port set. In order for indexer discovery to work, all of the indexers must have a receiving port set - the master node collects this information and then supplies it to the forwarder.

It looks like the forwarder connected to the master node correctly (your config files look fine) - but the master couldn't supply the requested information. The "hostport":"?" part of the message is what makes me believe that this is the problem.

View solution in original post

brent_weaver
Builder

Good morning, I did have one indexer that did not have the port listening and I had added the listener service (9997) and I still get the same message?!?!?

0 Karma

lguinn2
Legend

You might want to restart the master node.

0 Karma

lguinn2
Legend

I think this error message comes up when one of the indexers does not have a receiving port set. In order for indexer discovery to work, all of the indexers must have a receiving port set - the master node collects this information and then supplies it to the forwarder.

It looks like the forwarder connected to the master node correctly (your config files look fine) - but the master couldn't supply the requested information. The "hostport":"?" part of the message is what makes me believe that this is the problem.

brent_weaver
Builder

You my friend are an inspiration to people everywhere 🙂
THANK YOU for taking your valuable time to help me out, you were spot on!!!

0 Karma

lguinn2
Legend

(blush) You are welcome

0 Karma
Get Updates on the Splunk Community!

Set Up More Secure Configurations in Splunk Enterprise With Config Assist

This blog post is part 3 of 4 of a series on Splunk Assist. Click the links below to see the other ...

Observability Highlights | November 2022 Newsletter

 November 2022Observability CloudEnd Of Support Extension for SignalFx Smart AgentSplunk is extending the End ...

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...