Deployment Architecture

Getting decryption failed errors on indexers

VatsalJagani
SplunkTrust
SplunkTrust

In Splunk clustering, all the indexers are generating decryption failure errors in the splunkd (_internal) logs.

Crypto - Decryption operation failed: AES-GCM Decryption failed!
AesGcm - AES-GCM Decryption failed!

 

What could be the root cause and what is the solution? 

Labels (1)

jkat54
SplunkTrust
SplunkTrust

I had the same issue after copying a known working server.conf file (with encrypted pass4symmkey) to new peer, and once i had it on the new peer I updated the password (so that it wasnt encrypted) and saved.  

 

BUT then i realized it was in "DOS" format, and I used vim command ":set ff=unix" and then saved.  Switching from DOS to UNIX file format is what mangled it so that it could no longer be used.

To resolve, i opened server.conf and removed the encrypted pass4symmkey and replace it with the unencrypted value and restarted.

0 Karma

leeraym
Path Finder

I got the "AES-GCM decryption failed!" error on my search head after migrating from an old server to a new server (copying over the entire contents of $SPLUNK_HOME).  The fix that worked for me was to reset the pass4SymmKey in the [general] stanza of my $SPLUNK_HOME/etc/system/local/server.conf.  That key seems to affect a lot of things.  I had trouble sending email alerts and reading certain files with apps from SplunkBase until I corrected this.

1) On my old server, I ran $SPLUNK_HOME/bin/splunk show-decrypted --value '< pass4SymmKey value from server.conf>' in order to get the key in plaintext (requires Splunk 7.2.2+).

2) Then I edited $SPLUNK_HOME/etc/system/local/server.conf on my new server to set pass4SymmKey of the [general] stanza to the plaintext value from step 1.  For example, if your key from step 1 was "changeme", then change server.conf to look like this:

[general]
pass4SymmKey = changeme

3) Restart Splunk.  After Splunk starts, it will change the plaintext pass4SymmKey in your server.conf to an encrypted value.

4) Monitor $SPLUNK_HOME/var/log/splunk/splunkd.log and $SPLUNK_HOME/var/log/splunk/python.log to ensure that you don't get any more of those decryption failed messages.

rnowitzki
Builder

Hi @VatsalJagani ,

Someone spent a week solving this particular issue.

https://www.gnzlabs.io/gnzlabs-blog/splunk-aes-gcm-decryption-failed/

There is a solution you can try in the first few sentences, and an interesting crime story how he/she found the RootCause.

Did you recently upgrade your Splunk Env.?

--
Karma and/or Solution tagging appreciated.
0 Karma

coreyCLI
Path Finder

here is what i did to correct my issue with the same error at restart.  I was having the issue on 2 indexers.  I went throught the each value for every instance of pass4SymmKey and sslPassword and fed that encrypted value in the /bin/splunk show-decrypted --value 'pastvaluehere' until this command spit out the same error "....AES-GCM Decryption Failed!".  If it spits out the decrypted value, move on the the next one.  When that command spits out the AES-GCM error you know you have found your password that needs to be update/chagned.

Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...