ClusterManager Peer connection failed

[Mirza_Jaffar1]

what does indicates

06-19-2025 11:09:33.046 +0000 ERROR AesGcm [65605 MainThread] - Text decryption - error in finalizing: No errors in queue
06-19-2025 11:09:33.046 +0000 ERROR AesGcm [65605 MainThread] - AES-GCM Decryption failed!
06-19-2025 11:09:33.047 +0000 ERROR Crypto [65605 MainThread] - Decryption operation failed: AES-GCM Decryption failed!
06-19-2025 11:09:33.081 +0000 ERROR AesGcm [65605 MainThread] - Text decryption - error in finalizing: No errors in queue
06-19-2025 11:09:33.081 +0000 ERROR AesGcm [65605 MainThread] - AES-GCM Decryption failed!
06-19-2025 11:09:33.081 +0000 ERROR Crypto [65605 MainThread] - Decryption operation failed: AES-GCM Decryption failed!

[livehybrid]

Hi @Mirza_Jaffar1 

There is no mention of SSL in the error logs so I am leaning towards an issue with the pass4SymmKey or another encrypted credential. Have you recently made any changes or installed any apps?

If you copied a local directory from another instance that contained encrypted credentials then this instance will be unable to decrypt them, this is because Splunk encrypts credentials based on its own splunk.secret file 

You can verify encrypted keys such as pass4SymmKey by using:

$SPLUNK_HOME/bin/splunk show-decrypted --value '<value>'

When using this you need to change the $ -> \$ otherwise Linux will think this is a variable. for example $7$abc -> \$7\$abc

Please let us know what your architecture is like, e.g. what instance is this within your architecture and if you made any recent changes.

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[Mirza_Jaffar1]

can you please validate the below what could be issues as pe the config

splunk@mc1:/opt/splunk/etc/apps/ci1_unhash_app/local$ /opt/splunk/bin/splunk _internal call /storage/passwords/test

QUERYING: 'https://127.0.0.1:8089/services/storage/passwords/test'
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

Your session is invalid. Please login.
Splunk username: admin
Password:
FAILED: 'HTTP/1.1 404 Not Found'
Content:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="ERROR">Could not find object id=:test:</msg>
</messages>
</response>

splunk@mc1:/opt/splunk/etc/apps/ci1_unhash_app/local$ ll
total 16
drwxrwxr-x 2 splunk splunk 4096 Dec 8 18:53 ./
drwxrwxr-x 4 splunk splunk 4096 Dec 8 18:36 ../
-rw-rw-r-- 1 splunk splunk 110 Dec 8 18:19 app.conf
-rw-rw-r-- 1 splunk splunk 91 Dec 8 18:53 passwords.conf
splunk@ci1-persn000000001356580-mc1:/opt/splunk/etc/apps/ci1_unhash_app/local$ cat passwords.conf
[credential::test:]
password = $7$N/ZmtDftfjp7/ij6VGZeXh1l3UU2T6Ve+Hem3JCNna6upxmTvMDjSi==
splunk@mc1:/opt/splunk/etc/apps/ci1_unhash_app/local$

[PickleRick]

Actually, you need to escape the dollar sign if you are not using single quotes in most shells. If you are using single quotes for strings you should not escape the contents.

:/ $ echo \$
$
:/ $ echo "\$"
$
:/ $ echo '$'
$
:/ $ echo '\$'
\$

[livehybrid]

Nice thanks @PickleRick  - I rarely use single quotes with $ in so had assumed incorrectly it was the same as double quotes. 

Every day is a school day 🙂

Will

[Mirza_Jaffar1]

1. yes the local directory was copied from another instance
2. Tried to sync the directory from instance idx old to instance idx new
3. There seem some permission issues during the migration

[livehybrid]

Hi @Mirza_Jaffar1 

Did you copy the $SPLUNK_HOME/etc/auth/splunk.secret file from the old to the new server? This is the file that Splunk uses for encrypting sensitive configuration/secrets and is unique to each server, unless copied.

Regarding the permissions issues, did you manage to resolve these? Who are the the files/folders owned by and what user is the Splunk service running as?

🌟 Did this answer help you? If so, please consider:

• Adding karma to show it was useful
• Marking it as the solution if it resolved your issue
• Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

[PickleRick]

@Mirza_Jaffar1 Ok, so you tried to copy over the contents of old server's config to a new one, right? There were "some permission issues", right? Did you bother to check what kind of issues they were? Did you fix them?

[Mirza_Jaffar1]

I did check but nothing seems worked because chmod 770 is what used but chmod 550 should work! This something when usually occurs with permission.

Is there any other chmod numeric(550.775,7770) which provide same permission to the root and user?