Error updating FIPS compliance settings, during ES 6.1.0 upgrade

[pellegrini]

We get FIPS compliance error when upgrading to Enterprise Security 6.1.0.
FIPS is not enabled in our environment.

From start using Enterprise 7.1.2 and ES 5.3.0.
Upgrade to Enterprise 8.0.2.1 first, and then upgrade to ES 6.1.0. (This path should be supported as we understand)

-bash-4.2$ splunk show fips-mode -auth admin:passwd
FIPS mode disabled.

Splunk Enterprise Security Post-Install Configuration
When step 4.4 is running, we get error:
https://docs.splunk.com/Documentation/ES/6.1.0/Install/Upgradetonewerversion

Error in 'essinstall' command: postinstall failed - Error updating FIPS compliance settings. See search.log for details.

Extract from Search.log below

04-01-2020 16:56:21.052 ERROR ChunkedExternProcessor - stderr: msg="Error updating FIPS compliance settings."
04-01-2020 16:56:21.052 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last):
04-01-2020 16:56:21.052 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 142, in deployFips
04-01-2020 16:56:21.052 ERROR ChunkedExternProcessor - stderr:     incident_review_lookup_empty = isLookupEmpty(IR_LOOKUP, IR_APP, DEFAULT_OWNER, key)
04-01-2020 16:56:21.052 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 74, in isLookupEmpty
04-01-2020 16:56:21.052 ERROR ChunkedExternProcessor - stderr:     for lineno, unused_line in enumerate(open(path, 'r', newline=None)):
04-01-2020 16:56:21.052 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/lib/python3.7/codecs.py", line 322, in decode
04-01-2020 16:56:21.052 ERROR ChunkedExternProcessor - stderr:     (result, consumed) = self._buffer_decode(data, self.errors, final)
04-01-2020 16:56:21.052 ERROR ChunkedExternProcessor - stderr: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 37: invalid start byte
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr: 
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last):
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 142, in deployFips
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:     incident_review_lookup_empty = isLookupEmpty(IR_LOOKUP, IR_APP, DEFAULT_OWNER, key)
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 74, in isLookupEmpty
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:     for lineno, unused_line in enumerate(open(path, 'r', newline=None)):
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/lib/python3.7/codecs.py", line 322, in decode
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:     (result, consumed) = self._buffer_decode(data, self.errors, final)
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr: UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 37: invalid start byte
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr: During handling of the above exception, another exception occurred:
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last):
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/essinstaller2.py", line 331, in _postinstall
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:     deployFips(session_key, logger=self.logger)
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 165, in deployFips
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:     raise Exception('Error updating FIPS compliance settings.')
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr: Exception: Error updating FIPS compliance settings.
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr: During handling of the above exception, another exception occurred:
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr: Traceback (most recent call last):
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/essinstall.py", line 243, in do_install
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:     output = fn(session_key, True)
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/essinstaller2.py", line 81, in wrapper
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:     r = f(self, *args, **kwargs)
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/essinstaller2.py", line 571, in stage_postinstall
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:     self._postinstall(session_key)
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/essinstaller2.py", line 335, in _postinstall
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr:     raise InstallException(str(e))
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - stderr: install.app_install_utils.InstallException: Error updating FIPS compliance settings.
04-01-2020 16:56:21.053 ERROR ChunkedExternProcessor - Error in 'essinstall' command: postinstall failed - Error updating FIPS compliance settings.

We have restarted, selected to enable all Technology Add-on and also disable them all. Error message is always the same.

Sample from essinstaller2.log which might give a hint:

2020-04-01 14:56:21,051+0000 ERROR pid=6614 tid=MainThread file=deploy_fips_compliant_settings.py:deployFips:164 | msg="Error updating FIPS compliance settings."
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 142, in deployFips
incident_review_lookup_empty = isLookupEmpty(IR_LOOKUP, IR_APP, DEFAULT_OWNER, key)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 74, in isLookupEmpty
for lineno, unused_line in enumerate(open(path, 'r', newline=None)):
File "/opt/splunk/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 37: invalid start byte
2020-04-01 14:56:21,052+0000 ERROR pid=6614 tid=MainThread file=essinstall.py:do_install:261 |
Traceback (most recent call last):
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 142, in deployFips
incident_review_lookup_empty = isLookupEmpty(IR_LOOKUP, IR_APP, DEFAULT_OWNER, key)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 74, in isLookupEmpty
for lineno, unused_line in enumerate(open(path, 'r', newline=None)):
File "/opt/splunk/lib/python3.7/codecs.py", line 322, in decode
(result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa3 in position 37: invalid start byte

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/essinstaller2.py", line 331, in _postinstall
deployFips(session_key, logger=self.logger)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py", line 165, in deployFips
raise Exception('Error updating FIPS compliance settings.')
Exception: Error updating FIPS compliance settings.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/essinstall.py", line 243, in do_install
output = fn(session_key, True)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/essinstaller2.py", line 81, in wrapper
r = f(self, *args, **kwargs)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/essinstaller2.py", line 571, in stage_postinstall
self._postinstall(session_key)
File "/opt/splunk/etc/apps/SplunkEnterpriseSecuritySuite/bin/install/essinstaller2.py", line 335, in _postinstall
raise InstallException(str(e))
install.app_install_utils.InstallException: Error updating FIPS compliance settings.

[behlkush]

We faced a similar issue while upgrading to 6.0.2. It turned out not because of FIPS, but due to the upgrade the SSL Cert was expired

Below logs are from mongod.log and not splunkd.log:

<TIMESTAMP> I CONTROL [signalProcessingThread] shutting down with code:0
<TIMESTAMP> W CONTROL [main] net.ssl.sslCipherConfig is deprecated. It will be removed in a future release.
<TIMESTAMP> F NETWORK [main] The provided SSL certificate is expired or not yet valid.
<TIMESTAMP> F - [main] Fatal Assertion 28652 at src/mongo/util/net/ssl_manager.cpp 1157

Fix to above is to rename the server.pem to server.pem.old and restart splunk and rerun the installation.

We were able to reach mongod.log because of KV Store error messages coming up in the SH.

Hope this helps someone spending 3-4 hours to fix such a trivial upgrade issue.

[pellegrini]

This error was caused by hidden files coming from a Mac OS used to modify add-ons. Issue was solved by doing below:

We added own logging in this code, row 70-71 and 74-75:

$ vi etc/apps/SplunkEnterpriseSecuritySuite/bin/install/deploy_fips_compliant_settings.py

68 def isLookupEmpty(lookup_name, namespace, owner, key):
69 transform = SplunkLookupTransform.get(SplunkLookupTransform.build_id(lookup_name, namespace, owner), sessionKey=key)
70 # Custom row below
71 logger.info(Custom log, variable transform = "%s"', transform)
72 # Path will be a full path.
73 path = SplunkLookupTableFile.get(SplunkLookupTableFile.build_id(transform.filename, namespace, owner), sessionKey=key).path
74 # Custom row below
75 logger.info(Custom log, variable path = "%s"', path)

When in Splunk Web starting "Splunk Enterprise Security Post-Install Configuration", we get the following additional logs:

2020-04-08 07:25:51,515+0000 INFO pid=28843 tid=MainThread file=deploy_fips_compliant_settings.py:isLookupEmpty:71 | Custom, variable transform = "Owner: nobody, Namespace: SA-ThreatIntelligence, Name: incident_review_lookup, Id: /servicesNS/nobody/SA-ThreatIntelligence/configs/conf-transforms/incident_review_lookup"

2020-04-08 07:25:51,550+0000 INFO pid=28843 tid=MainThread file=deploy_fips_compliant_settings.py:isLookupEmpty:75 | Custom, variable path = "/opt/splunk/etc/apps/TA-Exchange-Mailbox/lookups/._event_id_to_action.csv"

$ ll -a /opt/splunk/etc/apps/TA-Exchange-Mailbox/lookups
total 8
drwxr-xr-x 2 splunk splunk 43 Apr 8 09:48 .
drwxr-xr-x 8 splunk splunk 4096 Feb 17 14:10 ..
-rw-r--r-- 1 splunk splunk 94 Feb 17 14:10 event_id_to_action.csv
-rw-r--r-- 1 splunk splunk 213 Jun 5 2019 ._event_id_to_action.csv

$ strings /opt/splunk/etc/apps/TA-Exchange-Mailbox/lookups/._event_id_to_action.csv
Mac OS X
ATTR
com.apple.quarantine
q/0081;5d135815;Firefox;DD12DC91-402F-4761-B9D4-8A0CE05C22B6

There might be more ._ files so make sure you remove them all.

This file comes from the initial installation of ES where a Mac was used to prepare some config. This file should not exist in Splunk.
So we removed that file and retry "Splunk Enterprise Security Post-Install Configuration" in Splunk Web and post install is successful.