Why am I unable to connect my forwarder to an indexer cluster with error "failed to extract FwdTarget from json node..."?

[cwyse]

I'm trying to get my forwarder to connect to an indexer cluster. I've tried changing every possible instance of pass4SymmKey to the same thing, but I still keep seeing the following error:

ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=default-autolb-group, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK

If I change the key to something different on the cluster master and forwarder, I get an Unauthorized error, so I don't really think it's the key, but not sure what else would be causing this. Any ideas?

[brent_weaver]

I am having the same issue as you guys and it is driving me crazy. I have three env's and only one has this issue.

[vanallp]

I found my issues…
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Multisiteconffile
My script installs the master, indexers, and search machines it performs a restart on each of them. Once that is complete, I deploy a configuration bundle to the indexers. That was my earlier problem – I was not performing the restart on all the splunk instances after the initial install.

[mstephenson716]

I forgot to restart my indexers after making them receivers. Thank you!

[meleperuma]

Thanks @vanallp. this is exactly what I was facing and it worked.

[Topographic0cea · Accepted solution]

I finally figure this out. On one indexer I did not open the listen port with spunk enable listen 9997. Once I did this, the error went away and all worked fine. The error message is completely non-intuitive.

[MuS]

Just fixed the exact same error, but my fix was to change the inputs.conf from

 [splunktcp://:9997]

to

[splunktcp://9997]

cheers, MuS

[Topographic0cea]

I am seeing the same thing. it is NOT a password problem. If it was, you would get http_response=Unauthorized.

If I configure the forwarder to send to one indexer in the cluster, that works just fine. It is only if you try to use the cluster master to do indexer discovery. What the missing or bad config it, I cannot figure out.

[vanallp]

I'm having the same issue. I setup a test environment with 3 indexers in a cluster that worked fine. Now I am attempting to setup a pair of indexer clusters and the indexer discovery is failing.

[jkat54]

Hello, did you check the server.conf in both the local and default directories?

The pass4symkey gets encrypted in the server.conf when it is deployed. I believe it deploys to appName/default and encrypts to appName/local. Then when you later update this server.conf app and deploy the app, the deployment server overwrites what is in /default but not /local... so the old encrypted pass4symkey takes precedence.

So run this command and make sure the correct pass4symkey is being "read" and taking "precedence"

./splunk cmd btool server list --debug

http://docs.splunk.com/Documentation/Splunk/6.1/admin/Wheretofindtheconfigurationfiles