topic Re: Getting decryption failed errors on indexers in Deployment Architecture

Getting decryption failed errors on indexers

VatsalJagani — Sun, 12 Jul 2020 17:13:21 GMT

In Splunk clustering, all the indexers are generating decryption failure errors in the splunkd (_internal) logs.

Crypto - Decryption operation failed: AES-GCM Decryption failed! AesGcm - AES-GCM Decryption failed!

What could be the root cause and what is the solution?

Re: Getting decryption failed errors on indexers

rnowitzki — Mon, 13 Jul 2020 09:32:35 GMT

Hi @VatsalJagani ,

Someone spent a week solving this particular issue.

https://www.gnzlabs.io/gnzlabs-blog/splunk-aes-gcm-decryption-failed/

There is a solution you can try in the first few sentences, and an interesting crime story how he/she found the RootCause.

Did you recently upgrade your Splunk Env.?

Re: Getting decryption failed errors on indexers

leeraym — Mon, 26 Oct 2020 15:44:42 GMT

I got the "AES-GCM decryption failed!" error on my search head after migrating from an old server to a new server (copying over the entire contents of $SPLUNK_HOME). The fix that worked for me was to reset the pass4SymmKey in the [general] stanza of my $SPLUNK_HOME/etc/system/local/server.conf. That key seems to affect a lot of things. I had trouble sending email alerts and reading certain files with apps from SplunkBase until I corrected this.

1) On my old server, I ran $SPLUNK_HOME/bin/splunk show-decrypted --value '< pass4SymmKey value from server.conf>' in order to get the key in plaintext (requires Splunk 7.2.2+).

2) Then I edited $SPLUNK_HOME/etc/system/local/server.conf on my new server to set pass4SymmKey of the [general] stanza to the plaintext value from step 1. For example, if your key from step 1 was "changeme", then change server.conf to look like this:

[general]
pass4SymmKey = changeme

3) Restart Splunk. After Splunk starts, it will change the plaintext pass4SymmKey in your server.conf to an encrypted value.

4) Monitor $SPLUNK_HOME/var/log/splunk/splunkd.log and $SPLUNK_HOME/var/log/splunk/python.log to ensure that you don't get any more of those decryption failed messages.

Re: Getting decryption failed errors on indexers

coreyCLI — Mon, 16 Nov 2020 17:06:21 GMT

here is what i did to correct my issue with the same error at restart. I was having the issue on 2 indexers. I went throught the each value for every instance of pass4SymmKey and sslPassword and fed that encrypted value in the /bin/splunk show-decrypted --value 'pastvaluehere' until this command spit out the same error "....AES-GCM Decryption Failed!". If it spits out the decrypted value, move on the the next one. When that command spits out the AES-GCM error you know you have found your password that needs to be update/chagned.

Re: Getting decryption failed errors on indexers

jkat54 — Wed, 18 Aug 2021 20:31:50 GMT

I had the same issue after copying a known working server.conf file (with encrypted pass4symmkey) to new peer, and once i had it on the new peer I updated the password (so that it wasnt encrypted) and saved.

BUT then i realized it was in "DOS" format, and I used vim command ":set ff=unix" and then saved. Switching from DOS to UNIX file format is what mangled it so that it could no longer be used.

To resolve, i opened server.conf and removed the encrypted pass4symmkey and replace it with the unencrypted value and restarted.