Getting decryption failed errors on indexers

[VatsalJagani]

In Splunk clustering, all the indexers are generating decryption failure errors in the splunkd (_internal) logs.

Crypto - Decryption operation failed: AES-GCM Decryption failed!
AesGcm - AES-GCM Decryption failed!

What could be the root cause and what is the solution? 

[jkat54]

I had the same issue after copying a known working server.conf file (with encrypted pass4symmkey) to new peer, and once i had it on the new peer I updated the password (so that it wasnt encrypted) and saved.  

BUT then i realized it was in "DOS" format, and I used vim command ":set ff=unix" and then saved.  Switching from DOS to UNIX file format is what mangled it so that it could no longer be used.

To resolve, i opened server.conf and removed the encrypted pass4symmkey and replace it with the unencrypted value and restarted.

[leeraym]

I got the "AES-GCM decryption failed!" error on my search head after migrating from an old server to a new server (copying over the entire contents of $SPLUNK_HOME).  The fix that worked for me was to reset the pass4SymmKey in the [general] stanza of my $SPLUNK_HOME/etc/system/local/server.conf.  That key seems to affect a lot of things.  I had trouble sending email alerts and reading certain files with apps from SplunkBase until I corrected this.

1) On my old server, I ran $SPLUNK_HOME/bin/splunk show-decrypted --value '< pass4SymmKey value from server.conf>' in order to get the key in plaintext (requires Splunk 7.2.2+).

2) Then I edited $SPLUNK_HOME/etc/system/local/server.conf on my new server to set pass4SymmKey of the [general] stanza to the plaintext value from step 1.  For example, if your key from step 1 was "changeme", then change server.conf to look like this:

[general]
pass4SymmKey = changeme

3) Restart Splunk.  After Splunk starts, it will change the plaintext pass4SymmKey in your server.conf to an encrypted value.

4) Monitor $SPLUNK_HOME/var/log/splunk/splunkd.log and $SPLUNK_HOME/var/log/splunk/python.log to ensure that you don't get any more of those decryption failed messages.

[rnowitzki]

Hi @VatsalJagani ,

Someone spent a week solving this particular issue.

https://www.gnzlabs.io/gnzlabs-blog/splunk-aes-gcm-decryption-failed/

There is a solution you can try in the first few sentences, and an interesting crime story how he/she found the RootCause.

Did you recently upgrade your Splunk Env.?

[coreyCLI]

here is what i did to correct my issue with the same error at restart.  I was having the issue on 2 indexers.  I went throught the each value for every instance of pass4SymmKey and sslPassword and fed that encrypted value in the /bin/splunk show-decrypted --value 'pastvaluehere' until this command spit out the same error "....AES-GCM Decryption Failed!".  If it spits out the decrypted value, move on the the next one.  When that command spits out the AES-GCM error you know you have found your password that needs to be update/chagned.