Deployment Architecture

Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Explorer

Hi,
In one of our new servers (linux), we installed Splunk 7.2.1 and promoted it as a cluster master. After the install, the Splunk started fine but the web is unable to load. While restart/start of server, the highlighted message is shown. Is that the problem?

How to rectify the issue and how to bring the splunk web up?

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
Checking critical directories... Done
Checking indexes...
Validated: audit _internal _introspection _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
Done
Checking default conf files for edits...
Validating installed files against hashes from '/apps/splunk-master/splunk/splunk-7.2.1-be11b2c46e23-linux-2.6-x86
64-manifest'
All installed files intact.
Done
All preliminary checks passed.

thanks
Mehala

Splunk Employee
Splunk Employee

This error shows up when Splunk tries to decrypt something encrypted by another copy of Splunk.

Especially if you're a Splunk Cloud customer, make sure that you aren't copying add-ons from one forwarder to another. This can happen with the Universal Forwarder credentials app, or any other content that has encrypted credentials - the credentials get encrypted on first run, so copying to another forwarder will have it try to decrypt using an incorrect key.

Re-download your add-ons (from Splunkbase) and this should clear up.

0 Karma

Splunk Employee
Splunk Employee

I was getting a similar "ERROR Crypto - Decryption operation failed: AES-GCM Decryption failed!" error on a forwarder and it turned out the Splunk Add-on for Amazon Web Services (SplunkTAaws) https://splunkbase.splunk.com/app/1876/ wasn't configured properly.

0 Karma

Explorer

Hi Splunkers,

I was getting almost the same issue when I was trying to start my Splunk Search Head:

error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!

And I was able to fix it, by the following:

1) I have commented out the pass4SymmKey in /opt/splunk/etc/system/local/server.conf
[general]

pass4SymmKey = $........

2) I have changed the sslPassword to the Splunk default password:
sslPassword = password

3) After restart the server, the issue was no longer shown and Splunk automatically creates a new pass4SymmKey value.

Hope it works for anyone who needs to fix this issue!

Splunk Employee
Splunk Employee

This error message itself indicates splunk is having issue to decrypt encrypted parameters in system configuration files.
This can be from any system/local conf files.

What you want to do is to find all the encrypted password (begin with "$" in your conf files. (e.g. pass4Symmkey, sslPassword, etc), re-enter them in clear text format and restart splunk.

From Splunk version 7.2.2 and above, you may run below command to decrypt the encrypted password to find the original clear text password:
*./splunk show-decrypted --value < pass4SymmKey > *

https://docs.splunk.com/Documentation/Splunk/7.2.2/Security/ConfigureS2Sonnewcipher

If you're getting "Decryption operation failed: AES-GCM Decryption failed!" from running above decryption attempt on the same splunk instance, that means your encrypted key might be corrupt or incorrect and you should re-enter the correct clear text one.

Motivator

Thanks. For me the error occurred, when we migrate 'SplunkTAaws' addon to a new server. The encrypted password in the 'passwords.conf' seems corrupted. When we re-configured the 'Key ID' and 'Secret Key' in the 'passwords.conf' file, the error disappeared.

0 Karma

Explorer

We tried the ./splunk show-decrypted --value < pass4SymmKey > command but get no output. Is there anything we need to be aware of?

0 Karma

Splunk Employee
Splunk Employee

are you running Splunk version 7.2.2 or above? It's only introduced from 7.2.2.

0 Karma

Splunk Employee
Splunk Employee

Did the server instance start or ask for a new PEM key?
I suspect you edited the server.conf on this node and only edited the pass4SymmKey values but not the SSL field

Starting splunk server daemon (splunkd)...
Enter PEM pass phrase:

Verify your password and assure they are consistent.
Best of luck

0 Karma

Splunk Employee
Splunk Employee

In my test env I replicated the error.
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!

If you are not using SSL on the node or have disabled the webserver remove the SSL stanza from server.conf, restart and check the logs. The node will connect to the cluster master.
Best~

0 Karma

SplunkTrust
SplunkTrust

If someone else wonders how he or she got here. Sometimes you deploy an app/configuration that contains your pass4SymmKeys. However, etc/system/local has the highest precedence, you might have an auto-generated key in etc/system/local/server.conf which you didn't expect which screws your configuration.

Just came across this and wanted to share it. In case you get the same message "AES-GCM Decryption failed" - check your system/local.

Skalli

0 Karma

Communicator

try to curl to the first page using this:

curl -v -k https://localhost:8000
OR
curl -v http://localhost:8000
Otherwise, check to see if you have OpenSSL installed and what version it is.

Other than that, you should open a case with Splunk Support.

0 Karma