In Splunk clustering, all the indexers are generating decryption failure errors in the splunkd (_internal) logs.
Crypto - Decryption operation failed: AES-GCM Decryption failed! AesGcm - AES-GCM Decryption failed!
What could be the root cause and what is the solution?
I had the same issue after copying a known working server.conf file (with encrypted pass4symmkey) to new peer, and once i had it on the new peer I updated the password (so that it wasnt encrypted) and saved.
BUT then i realized it was in "DOS" format, and I used vim command ":set ff=unix" and then saved. Switching from DOS to UNIX file format is what mangled it so that it could no longer be used.
To resolve, i opened server.conf and removed the encrypted pass4symmkey and replace it with the unencrypted value and restarted.
I got the "AES-GCM decryption failed!" error on my search head after migrating from an old server to a new server (copying over the entire contents of $SPLUNK_HOME). The fix that worked for me was to reset the pass4SymmKey in the [general] stanza of my $SPLUNK_HOME/etc/system/local/server.conf. That key seems to affect a lot of things. I had trouble sending email alerts and reading certain files with apps from SplunkBase until I corrected this.
1) On my old server, I ran $SPLUNK_HOME/bin/splunk show-decrypted --value '< pass4SymmKey value from server.conf>' in order to get the key in plaintext (requires Splunk 7.2.2+).
2) Then I edited $SPLUNK_HOME/etc/system/local/server.conf on my new server to set pass4SymmKey of the [general] stanza to the plaintext value from step 1. For example, if your key from step 1 was "changeme", then change server.conf to look like this:
pass4SymmKey = changeme
3) Restart Splunk. After Splunk starts, it will change the plaintext pass4SymmKey in your server.conf to an encrypted value.
4) Monitor $SPLUNK_HOME/var/log/splunk/splunkd.log and $SPLUNK_HOME/var/log/splunk/python.log to ensure that you don't get any more of those decryption failed messages.
Hi @VatsalJagani ,
Someone spent a week solving this particular issue.
There is a solution you can try in the first few sentences, and an interesting crime story how he/she found the RootCause.
Did you recently upgrade your Splunk Env.?
here is what i did to correct my issue with the same error at restart. I was having the issue on 2 indexers. I went throught the each value for every instance of pass4SymmKey and sslPassword and fed that encrypted value in the /bin/splunk show-decrypted --value 'pastvaluehere' until this command spit out the same error "....AES-GCM Decryption Failed!". If it spits out the decrypted value, move on the the next one. When that command spits out the AES-GCM error you know you have found your password that needs to be update/chagned.