Deployment Architecture
Highlighted

Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Explorer

Hi,
In one of our new servers (linux), we installed Splunk 7.2.1 and promoted it as a cluster master. After the install, the Splunk started fine but the web is unable to load. While restart/start of server, the highlighted message is shown. Is that the problem?

How to rectify the issue and how to bring the splunk web up?

Checking prerequisites...
Checking http port [8000]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
Checking critical directories... Done
Checking indexes...
Validated: audit _internal _introspection _telemetry _thefishbucket history main summary
Done
Checking filesystem compatibility... Done
Checking conf files for problems...
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
Done
Checking default conf files for edits...
Validating installed files against hashes from '/apps/splunk-master/splunk/splunk-7.2.1-be11b2c46e23-linux-2.6-x86
64-manifest'
All installed files intact.
Done
All preliminary checks passed.

thanks
Mehala

Highlighted

Re: Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Communicator

try to curl to the first page using this:

curl -v -k https://localhost:8000
OR
curl -v http://localhost:8000
Otherwise, check to see if you have OpenSSL installed and what version it is.

Other than that, you should open a case with Splunk Support.

0 Karma
Highlighted

Re: Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Splunk Employee
Splunk Employee

Did the server instance start or ask for a new PEM key?
I suspect you edited the server.conf on this node and only edited the pass4SymmKey values but not the SSL field

Starting splunk server daemon (splunkd)...
Enter PEM pass phrase:

Verify your password and assure they are consistent.
Best of luck

0 Karma
Highlighted

Re: Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Splunk Employee
Splunk Employee

In my test env I replicated the error.
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!
error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!

If you are not using SSL on the node or have disabled the webserver remove the SSL stanza from server.conf, restart and check the logs. The node will connect to the cluster master.
Best~

0 Karma
Highlighted

Re: Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

SplunkTrust
SplunkTrust

If someone else wonders how he or she got here. Sometimes you deploy an app/configuration that contains your pass4SymmKeys. However, etc/system/local has the highest precedence, you might have an auto-generated key in etc/system/local/server.conf which you didn't expect which screws your configuration.

Just came across this and wanted to share it. In case you get the same message "AES-GCM Decryption failed" - check your system/local.

Skalli

0 Karma
Highlighted

Re: Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Splunk Employee
Splunk Employee

This error message itself indicates splunk is having issue to decrypt encrypted parameters in system configuration files.
This can be from any system/local conf files.

What you want to do is to find all the encrypted password (begin with "$" in your conf files. (e.g. pass4Symmkey, sslPassword, etc), re-enter them in clear text format and restart splunk.

From Splunk version 7.2.2 and above, you may run below command to decrypt the encrypted password to find the original clear text password:
*./splunk show-decrypted --value < pass4SymmKey > *

https://docs.splunk.com/Documentation/Splunk/7.2.2/Security/ConfigureS2Sonnewcipher

If you're getting "Decryption operation failed: AES-GCM Decryption failed!" from running above decryption attempt on the same splunk instance, that means your encrypted key might be corrupt or incorrect and you should re-enter the correct clear text one.

Highlighted

Re: Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Explorer

We tried the ./splunk show-decrypted --value < pass4SymmKey > command but get no output. Is there anything we need to be aware of?

0 Karma
Highlighted

Re: Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Splunk Employee
Splunk Employee

are you running Splunk version 7.2.2 or above? It's only introduced from 7.2.2.

0 Karma
Highlighted

Re: Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Motivator

Thanks. For me the error occurred, when we migrate 'SplunkTAaws' addon to a new server. The encrypted password in the 'passwords.conf' seems corrupted. When we re-configured the 'Key ID' and 'Secret Key' in the 'passwords.conf' file, the error disappeared.

0 Karma
Highlighted

Re: Installed Splunk Enterprise 7.2.1 and promoted it as cluster master. Splunk Web is not loading.

Explorer

Hi Splunkers,

I was getting almost the same issue when I was trying to start my Splunk Search Head:

error:00000000:lib(0):func(0):reason(0)
AES-GCM Decryption failed!
Decryption operation failed: AES-GCM Decryption failed!

And I was able to fix it, by the following:

1) I have commented out the pass4SymmKey in /opt/splunk/etc/system/local/server.conf
[general]

pass4SymmKey = $........

2) I have changed the sslPassword to the Splunk default password:
sslPassword = password

3) After restart the server, the issue was no longer shown and Splunk automatically creates a new pass4SymmKey value.

Hope it works for anyone who needs to fix this issue!