Splunk web fails to start after upgrade to 8.0.1 - UnicodeDecodeError: 'ascii' codec can't decode

[srauhala_splunk]

After upgrading from SE 7.3 to SE 8.0.1 Splunk web failed to start

[splunk@ip-10-202-18-65 ~]$ /opt/splunk/bin/splunk start
The splunk daemon (splunkd) is already running. [MISSLYCKADES]

Waiting for web server at http://127.0.0.1:8000 to be available............................................................................................................................................................................................................................................................................................................

WARNING: web interface does not seem to be available!

[splunk@ip-10-202-18-65 ~]$ cat /opt/splunk/var/log/splunk/web_service.log
2020-03-11 14:13:29,700 ERROR [5e68f209517f57edcf17d0] root:770 - 'ascii' codec can't decode byte 0xc3 in position 17: ordinal not in range(128)
Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 132, in
from splunk.appserver.mrsparkle.controllers.top import TopController
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/top.py", line 27, in
from splunk.appserver.mrsparkle.controllers.admin import AdminController
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 17, in
import formencode
File "/opt/splunk/lib/python3.7/site-packages/formencode/init.py", line 3, in
from formencode.api import (
File "/opt/splunk/lib/python3.7/site-packages/formencode/api.py", line 65, in
set_stdtranslation()
File "/opt/splunk/lib/python3.7/site-packages/formencode/api.py", line 58, in set_stdtranslation
localedir=localedir, fallback=True)
File "/opt/splunk/lib/python3.7/gettext.py", line 533, in translation
t = translations.setdefault(key, class(fp))
File "/opt/splunk/lib/python3.7/gettext.py", line 260, in init
self.parse(fp)
File "/opt/splunk/lib/python3.7/gettext.py", line 421, in _parse
catalog[str(msg, charset)] = str(tmsg, charset)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 17: ordinal not in range(128)
2020-03-11 14:35:51,498 INFO [5e68f747617fbbe19b7f90] __init:174 - Using default logging config file: /opt/splunk/etc/log.cfg
2020-03-11 14:35:51,499 INFO [5e68f747617fbbe19b7f90] __init:212 - Setting logger=splunk level=INFO
2020-03-11 14:35:51,499 INFO [5e68f747617fbbe19b7f90] __init:212 - Setting logger=splunk.appserver level=INFO
2020-03-11 14:35:51,499 INFO [5e68f747617fbbe19b7f90] __init:212 - Setting logger=splunk.appserver.controllers level=INFO
2020-03-11 14:35:51,499 INFO [5e68f747617fbbe19b7f90] __init:212 - Setting logger=splunk.appserver.controllers.proxy level=INFO
2020-03-11 14:35:51,499 INFO [5e68f747617fbbe19b7f90] __init:212 - Setting logger=splunk.appserver.lib level=WARN
2020-03-11 14:35:51,499 INFO [5e68f747617fbbe19b7f90] __init:212 - Setting logger=splunk.pdfgen level=INFO
2020-03-11 14:35:51,499 INFO [5e68f747617fbbe19b7f90] __init:212 - Setting logger=splunk.archiver_restoration level=INFO
2020-03-11 14:35:51,551 ERROR [5e68f747617fbbe19b7f90] root:769 - Unable to start splunkweb
2020-03-11 14:35:51,551 ERROR [5e68f747617fbbe19b7f90] root:770 - 'ascii' codec can't decode byte 0xc3 in position 17: ordinal not in range(128)
Traceback (most recent call last):
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/root.py", line 132, in
from splunk.appserver.mrsparkle.controllers.top import TopController
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/top.py", line 27, in
from splunk.appserver.mrsparkle.controllers.admin import AdminController
File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/admin.py", line 17, in
import formencode
File "/opt/splunk/lib/python3.7/site-packages/formencode/init.py", line 3, in
from formencode.api import (
File "/opt/splunk/lib/python3.7/site-packages/formencode/api.py", line 65, in
set_stdtranslation()
File "/opt/splunk/lib/python3.7/site-packages/formencode/api.py", line 58, in set_stdtranslation
localedir=localedir, fallback=True)
File "/opt/splunk/lib/python3.7/gettext.py", line 533, in translation
t = _translations.setdefault(key, class(fp))
File "/opt/splunk/lib/python3.7/gettext.py", line 260, in init
self._parse(fp)
File "/opt/splunk/lib/python3.7/gettext.py", line 421, in _parse
catalog[str(msg, charset)] = str(tmsg, charset)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 17: ordinal not in range(128)

[srauhala_splunk]

To answer my own question, our problem was that the user running the splunk process had the locale sv_SE. This cased python3 to fail when importing formecode.

Workaround is to set the system and user locale to en_GB or en_US

[splunk@ip-10-202-18-65 ~]$ LANG=en_US.utf8
[splunk@ip-10-202-18-65 ~]$ /opt/splunk/bin/splunk start

[YoungDaniel]

Works! I adde the LANG var to the splunk-launcher script. It also does the job and help out for future restart.