topic Re: ClusterManager Peer connection failed in Splunk Enterprise Security

ClusterManager Peer connection failed

Mirza_Jaffar1 — Thu, 19 Jun 2025 11:32:25 GMT

what does indicates

06-19-2025 11:09:33.046 +0000 ERROR AesGcm [65605 MainThread] - Text decryption - error in finalizing: No errors in queue
06-19-2025 11:09:33.046 +0000 ERROR AesGcm [65605 MainThread] - AES-GCM Decryption failed!
06-19-2025 11:09:33.047 +0000 ERROR Crypto [65605 MainThread] - Decryption operation failed: AES-GCM Decryption failed!
06-19-2025 11:09:33.081 +0000 ERROR AesGcm [65605 MainThread] - Text decryption - error in finalizing: No errors in queue
06-19-2025 11:09:33.081 +0000 ERROR AesGcm [65605 MainThread] - AES-GCM Decryption failed!
06-19-2025 11:09:33.081 +0000 ERROR Crypto [65605 MainThread] - Decryption operation failed: AES-GCM Decryption failed!

Re: ClusterManager Peer connection failed

livehybrid — Thu, 19 Jun 2025 12:35:36 GMT

Hi @Mirza_Jaffar1

There is no mention of SSL in the error logs so I am leaning towards an issue with the pass4SymmKey or another encrypted credential. Have you recently made any changes or installed any apps?

If you copied a local directory from another instance that contained encrypted credentials then this instance will be unable to decrypt them, this is because Splunk encrypts credentials based on its own splunk.secret file

You can verify encrypted keys such as pass4SymmKey by using:

$SPLUNK_HOME/bin/splunk show-decrypted --value '<value>'

When using this you need to change the $ -> \$ otherwise Linux will think this is a variable. for example $7$abc -> \$7\$abc

Please let us know what your architecture is like, e.g. what instance is this within your architecture and if you made any recent changes.

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: ClusterManager Peer connection failed

Mirza_Jaffar1 — Mon, 14 Jul 2025 21:04:54 GMT

yes the local directory was copied from another instance
Tried to sync the directory from instance idx old to instance idx new
There seem some permission issues during the migration

Re: ClusterManager Peer connection failed

PickleRick — Mon, 14 Jul 2025 21:30:45 GMT

Actually, you need to escape the dollar sign if you are not using single quotes in most shells. If you are using single quotes for strings you should not escape the contents.

:/ $ echo \$ $ :/ $ echo "\$" $ :/ $ echo '$' $ :/ $ echo '\$' \$

Re: ClusterManager Peer connection failed

PickleRick — Mon, 14 Jul 2025 21:33:05 GMT

@Mirza_Jaffar1 Ok, so you tried to copy over the contents of old server's config to a new one, right? There were "some permission issues", right? Did you bother to check what kind of issues they were? Did you fix them?

Re: ClusterManager Peer connection failed

Mirza_Jaffar1 — Mon, 14 Jul 2025 21:37:44 GMT

I did check but nothing seems worked because chmod 770 is what used but chmod 550 should work! This something when usually occurs with permission.

Is there any other chmod numeric(550.775,7770) which provide same permission to the root and user?

Re: ClusterManager Peer connection failed

livehybrid — Mon, 14 Jul 2025 21:38:57 GMT

Hi @Mirza_Jaffar1

Did you copy the $SPLUNK_HOME/etc/auth/splunk.secret file from the old to the new server? This is the file that Splunk uses for encrypting sensitive configuration/secrets and is unique to each server, unless copied.

Regarding the permissions issues, did you manage to resolve these? Who are the the files/folders owned by and what user is the Splunk service running as?

🌟 Did this answer help you? If so, please consider:

Adding karma to show it was useful
Marking it as the solution if it resolved your issue
Commenting if you need any clarification

Your feedback encourages the volunteers in this community to continue contributing

Re: ClusterManager Peer connection failed

livehybrid — Mon, 14 Jul 2025 21:39:54 GMT

Nice thanks @PickleRick - I rarely use single quotes with $ in so had assumed incorrectly it was the same as double quotes.

Every day is a school day 🙂

Will

Re: ClusterManager Peer connection failed

Mirza_Jaffar1 — Mon, 08 Dec 2025 19:18:02 GMT

can you please validate the below what could be issues as pe the config

splunk@mc1:/opt/splunk/etc/apps/ci1_unhash_app/local$ /opt/splunk/bin/splunk _internal call /storage/passwords/test

QUERYING: 'https://127.0.0.1:8089/services/storage/passwords/test'
WARNING: Server Certificate Hostname Validation is disabled. Please see server.conf/[sslConfig]/cliVerifyServerName for details.

Your session is invalid. Please login.
Splunk username: admin
Password:
FAILED: 'HTTP/1.1 404 Not Found'
Content:
<?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="ERROR">Could not find object id=:test:</msg>
</messages>
</response>

splunk@mc1:/opt/splunk/etc/apps/ci1_unhash_app/local$ ll
total 16
drwxrwxr-x 2 splunk splunk 4096 Dec 8 18:53 ./
drwxrwxr-x 4 splunk splunk 4096 Dec 8 18:36 ../
-rw-rw-r-- 1 splunk splunk 110 Dec 8 18:19 app.conf
-rw-rw-r-- 1 splunk splunk 91 Dec 8 18:53 passwords.conf
splunk@ci1-persn000000001356580-mc1:/opt/splunk/etc/apps/ci1_unhash_app/local$ cat passwords.conf
[credential::test:]
password = $7$N/ZmtDftfjp7/ij6VGZeXh1l3UU2T6Ve+Hem3JCNna6upxmTvMDjSi==
splunk@mc1:/opt/splunk/etc/apps/ci1_unhash_app/local$