topic Re: Why am I unable to get index discovery working with ID? in Deployment Architecture

Why am I unable to get index discovery working with ID?

brent_weaver — Sat, 03 Feb 2018 13:21:55 GMT

I am using index discovery in 2 of my 3 Splunk env's and I have one that will simply not work with ID. I get the following error:

02-03-2018 13:14:23.597 +0000 ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=group1, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK

I have NO IDEA why this is happening, the same automation builds this env as the others.

This is my client config:

[indexer_discovery:idx]
pass4SymmKey = MYSECRET
master_uri = https://myhost:8089

[tcpout:group1]
indexerDiscovery = idx

[tcpout]
defaultGroup = group1

Here is the master index server config:

[indexer_discovery]
pass4SymmKey = MYSECRETKEY
polling_rate = 10
indexerWeightByDiskCapacity = 0

Of course, the keys are hashed. I really hope that someone can figure this one out! I have no clue as it seems like Splunk cannot parse the file or params are missing. I can connect to the master index server from the peers on 8089 without a problem!

Re: Why am I unable to get index discovery working with ID?

lguinn2 — Mon, 05 Feb 2018 20:50:23 GMT

I think this error message comes up when one of the indexers does not have a receiving port set. In order for indexer discovery to work, all of the indexers must have a receiving port set - the master node collects this information and then supplies it to the forwarder.

It looks like the forwarder connected to the master node correctly (your config files look fine) - but the master couldn't supply the requested information. The "hostport":"?" part of the message is what makes me believe that this is the problem.

Re: Why am I unable to get index discovery working with ID?

brent_weaver — Mon, 05 Feb 2018 21:09:30 GMT

You my friend are an inspiration to people everywhere 🙂
THANK YOU for taking your valuable time to help me out, you were spot on!!!

Re: Why am I unable to get index discovery working with ID?

lguinn2 — Tue, 06 Feb 2018 01:43:11 GMT

(blush) You are welcome

Re: Why am I unable to get index discovery working with ID?

brent_weaver — Tue, 06 Feb 2018 13:12:58 GMT

Good morning, I did have one indexer that did not have the port listening and I had added the listener service (9997) and I still get the same message?!?!?

Re: Why am I unable to get index discovery working with ID?

lguinn2 — Fri, 09 Feb 2018 23:23:37 GMT

You might want to restart the master node.