Getting Data In

Why am I unable to connect my forwarder to an indexer cluster with error "failed to extract FwdTarget from json node..."?

cwyse
Explorer

I'm trying to get my forwarder to connect to an indexer cluster. I've tried changing every possible instance of pass4SymmKey to the same thing, but I still keep seeing the following error:

ERROR IndexerDiscoveryHeartbeatThread - failed to parse response payload for group=default-autolb-group, err=failed to extract FwdTarget from json node={"hostport":"?","ssl":false,"indexing_disk_space":-1}http_response=OK

If I change the key to something different on the cluster master and forwarder, I get an Unauthorized error, so I don't really think it's the key, but not sure what else would be causing this. Any ideas?

1 Solution

Topographic0cea
Explorer

I finally figure this out. On one indexer I did not open the listen port with spunk enable listen 9997. Once I did this, the error went away and all worked fine. The error message is completely non-intuitive.

View solution in original post

brent_weaver
Builder

I am having the same issue as you guys and it is driving me crazy. I have three env's and only one has this issue.

0 Karma

vanallp
Explorer

I found my issues…
http://docs.splunk.com/Documentation/Splunk/latest/Indexer/Multisiteconffile
My script installs the master, indexers, and search machines it performs a restart on each of them. Once that is complete, I deploy a configuration bundle to the indexers. That was my earlier problem – I was not performing the restart on all the splunk instances after the initial install.

mstephenson716
Explorer

I forgot to restart my indexers after making them receivers. Thank you!

0 Karma

meleperuma
Explorer

Thanks @vanallp. this is exactly what I was facing and it worked.

0 Karma

Topographic0cea
Explorer

I finally figure this out. On one indexer I did not open the listen port with spunk enable listen 9997. Once I did this, the error went away and all worked fine. The error message is completely non-intuitive.

MuS
SplunkTrust
SplunkTrust

Just fixed the exact same error, but my fix was to change the inputs.conf from

 [splunktcp://:9997]

to

[splunktcp://9997]

cheers, MuS

Topographic0cea
Explorer

I am seeing the same thing. it is NOT a password problem. If it was, you would get http_response=Unauthorized.

If I configure the forwarder to send to one indexer in the cluster, that works just fine. It is only if you try to use the cluster master to do indexer discovery. What the missing or bad config it, I cannot figure out.

0 Karma

vanallp
Explorer

I'm having the same issue. I setup a test environment with 3 indexers in a cluster that worked fine. Now I am attempting to setup a pair of indexer clusters and the indexer discovery is failing.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Hello, did you check the server.conf in both the local and default directories?

The pass4symkey gets encrypted in the server.conf when it is deployed. I believe it deploys to appName/default and encrypts to appName/local. Then when you later update this server.conf app and deploy the app, the deployment server overwrites what is in /default but not /local... so the old encrypted pass4symkey takes precedence.

So run this command and make sure the correct pass4symkey is being "read" and taking "precedence"

./splunk cmd btool server list --debug

http://docs.splunk.com/Documentation/Splunk/6.1/admin/Wheretofindtheconfigurationfiles

0 Karma
Get Updates on the Splunk Community!

Adoption of RUM and APM at Splunk

    Unleash the power of Splunk Observability   Watch Now In this can't miss Tech Talk! The Splunk Growth ...

March Community Office Hours Security Series Uncovered!

Hello Splunk Community! In March, Splunk Community Office Hours spotlighted our fabulous Splunk Threat ...

Stay Connected: Your Guide to April Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars in April. This post ...