Why is Splunk Web is not loading after Installing Splunk Enterprise 7.2.1 and promoting it as cluster master?

[mehala12]

Hi,
In one of our new servers (linux), we installed Splunk 7.2.1 and promoted it as a cluster master. After the install, the Splunk started fine but the web is unable to load. While restart/start of server, the highlighted message is shown. Is that the problem?

How to rectify the issue and how to bring the splunk web up?

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
Done
Checking default conf files for edits...
Validating installed files against hashes from '/apps/splunk-master/splunk/splunk-7.2.1-be11b2c46e23-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.

thanks
Mehala

[jkoby_splunk]

This error shows up when Splunk tries to decrypt something encrypted by another copy of Splunk.

Especially if you're a Splunk Cloud customer, make sure that you aren't copying add-ons from one forwarder to another. This can happen with the Universal Forwarder credentials app, or any other content that has encrypted credentials - the credentials get encrypted on first run, so copying to another forwarder will have it try to decrypt using an incorrect key.

Re-download your add-ons (from Splunkbase) and this should clear up.

[mmacvicar_splun]

I was getting a similar "ERROR Crypto - Decryption operation failed: AES-GCM Decryption failed!" error on a forwarder and it turned out the Splunk Add-on for Amazon Web Services (Splunk_TA_aws) https://splunkbase.splunk.com/app/1876/ wasn't configured properly.

[marend]

Hi Splunkers,

I was getting almost the same issue when I was trying to start my Splunk Search Head:

error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!

And I was able to fix it, by the following:

1) I have commented out the pass4SymmKey in /opt/splunk/etc/system/local/server.conf
[general]

pass4SymmKey = $........

2) I have changed the sslPassword to the Splunk default password:
sslPassword = password

3) After restart the server, the issue was no longer shown and Splunk automatically creates a new pass4SymmKey value.

Hope it works for anyone who needs to fix this issue!

[scheng_splunk]

This error message itself indicates splunk is having issue to decrypt encrypted parameters in system configuration files.
This can be from any system/local conf files.

What you want to do is to find all the encrypted password (begin with "$" in your conf files. (e.g. pass4Symmkey, sslPassword, etc), re-enter them in clear text format and restart splunk.

From Splunk version 7.2.2 and above, you may run below command to decrypt the encrypted password to find the original clear text password:
*./splunk show-decrypted --value < pass4SymmKey > *

https://docs.splunk.com/Documentation/Splunk/7.2.2/Security/ConfigureS2Sonnewcipher

If you're getting "Decryption operation failed: AES-GCM Decryption failed!" from running above decryption attempt on the same splunk instance, that means your encrypted key might be corrupt or incorrect and you should re-enter the correct clear text one.

[jawaharas]

Thanks. For me the error occurred, when we migrate 'Splunk_TA_aws' addon to a new server. The encrypted password in the 'passwords.conf' seems corrupted. When we re-configured the 'Key ID' and 'Secret Key' in the 'passwords.conf' file, the error disappeared.

[rasikmhetre]

hello, @jawaharas can you please tell me how you reconfigured the secret key in Splunk add on AWS, did you do it from web UI? I am struggling to do it from CLI, by directly inputting the values in inputs.conf and passwords.conf, because our indexer cluster does not open web.

[jawaharas]

Hi @rasikmhetre ,

You can add 'secret key' for the addon in the 'passwords.conf' file. The password will be encrypted once you restart the Splunk service. 

Can you explain the issue that you are facing with CLI?

[isoutamo]

Hi
It's best to create a new question for this than try to use couple of year old answer without solutions!

Try to use GUI for updating AWS accounts passwords. And your should run AWS TA on separate HF not in indexer cluster!

r. Ismo

[mspoerr]

We tried the ./splunk show-decrypted --value < pass4SymmKey > command but get no output. Is there anything we need to be aware of?

[scheng_splunk]

are you running Splunk version 7.2.2 or above? It's only introduced from 7.2.2.

[jetchings_splun]

Did the server instance start or ask for a new PEM key?
I suspect you edited the server.conf on this node and only edited the pass4SymmKey values but not the SSL field

Starting splunk server daemon (splunkd)...
Enter PEM pass phrase:

Verify your password and assure they are consistent.
Best of luck

[jetchings_splun]

In my test env I replicated the error.
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!

If you are not using SSL on the node or have disabled the webserver remove the SSL stanza from server.conf, restart and check the logs. The node will connect to the cluster master.
Best~

[skalliger]

If someone else wonders how he or she got here. Sometimes you deploy an app/configuration that contains your pass4SymmKeys. However, etc/system/local has the highest precedence, you might have an auto-generated key in etc/system/local/server.conf which you didn't expect which screws your configuration.

Just came across this and wanted to share it. In case you get the same message "AES-GCM Decryption failed" - check your system/local.

Skalli

[BainM]

try to curl to the first page using this:

curl -v -k https://localhost:8000
OR
curl -v http://localhost:8000
Otherwise, check to see if you have OpenSSL installed and what version it is.

Other than that, you should open a case with Splunk Support.