Why does Splunk assist constantly fail on 9.1.0.1 version after upgrade due to free license?

[ori]

hey, we're in the process of upgrading on our splunk single instances from 8.2.5 to 9.1.0.1 due to EOL.

we have two splunk instances - production which is under the enterprise license, test which is under the free license.

we've first started upgrading our test splunk instance with the free license and we instantly saw these errors:

07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py" [modular_input:349] [execute] [834704] Modular input: Splunk Assist exit with exception: Traceback (most recent call last):
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/assist/modular_input.py", line 342, in execute
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     self.do_run(input_definition["inputs"])
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py", line 66, in do_run
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     if not should_run(self.logger, self.session_key):
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py", line 27, in should_run
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     sh = is_search_head(log, session_key)
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/assist/serverinfo.py", line 153, in is_search_head
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     cluster_mode = get_cluster_mode(log, session_key)
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/assist/serverinfo.py", line 257, in get_cluster_mode
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     raiseAllErrors=True
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 646, in simpleRequest
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     raise splunk.LicenseRestriction
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py" splunk.LicenseRestriction: [HTTP 402] Current license does not allow the requested action
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py" .

 

this splunk assist issue seems to come from the usage of free license on splunk.

I tried to disable the splunk-assist app but it wouldn't let me:

Cannot disable app: splunk_assist

 
as a result of the previous errors, we are seeing UI errors as well: 

Unable to load common tasks. Refresh the page to try again.

any idea on how to proceed?

[yuanliu]

Free license or not free license, Splunk UI should not display cryptic error message.  I consider this a usability bug because the license type can easily be handled with clearer message.

You can disable Splunk Assist.  But that will not solve this problem.

To disable the application, go to $SPLUNK_HOME/etc/apps/splunk_assist.  Create a directory local/, then create a file app.conf to override allow_disable=true in [install] stanza, like this:

cd $SPLUNK_HOME/etc/apps/splunk_assist
mkdir local
cat <<EOM >local/app.conf
[install]
allows_disable = false
EOM

After restarting Splunk, you can then disable splunk_assist.  But the error will persist even when Splunk Assist is disabled.  BTW, if you launch splunk_assist directly, you will be able to see the error message

Unable to obtain template "beam:/templates/start.html": ... TopLevelLookupException(_("Splunk has failed to locate the template for uri '%s'." % uri)) TopLevelLookupException: Splunk has failed to locate the template for uri 'beam:/templates/start.html'.

The problem is that a 9.1 upgrade enables a feature called enable_home_vnext.  It is a fine feature (with tons of marketing plugs) except for this bug: Splunk should detect the free license and handle the error without showing a cryptic error message. (It can show, for example, "this feature is unavailable for the installed license" like it does with a bunch of other features.)

Workaround

The only workaround I have discovered so far is to disable enable_home_vnext in [feature:page_migration] stanza from web-features.conf.  ([feature:page_migration] is introduced in 9.x but is in web.conf. web-features.conf is introduced in 9.1.1; a skeleton local/web-features.conf is created by installer but the stanza is not in it.)

cd $SPLUNK_HOME/etc/system/local
cat <<EOM>>web-features.conf
[feature:page_migration]
enable_home_vnext = false
EOM

After this, you will see your previous launcher home page instead of the migration page. (You may need to to restart splunkd.  I had at least one instance for which I did not perform restart after editing.)  I hope Splunk will not take the legacy launch page away before fixing the bug in the new page.

What changed?

The new index _configtracker added in 9.x makes it much easier to track changes made over time.  Because I was upgrading from 9.0.5, I can see the exact changes 9.1 upgrade has made.  For example, to see web feature changes,

index="_configtracker" (data.path=*/system/default/web-features.conf)
| spath path=data.changes{}
| fields - data.changes{}.*
| mvexpand data.changes{}
| spath input=data.changes{}
| spath input=data.changes{} path=properties{}
| fields - properties{}.*
| mvexpand properties{}
| spath input=properties{}
| stats latest(*_value) as *_value by data.action name stanza data.path _time
| eval data.path = replace('data.path', ".*/[sS]plunk/etc", "")
| fieldformat _time = strftime(_time, "%F")
| table name *_value stanza data.* _time

On my laptop instance (defaults), this shows

name	new_value	old_value	stanza	data.action	data.path	_time
disable_highcharts_accessibility	false		feature:highcharts_accessibility	update	/system/default/web-features.conf	2023-09-08
enable_acuif_pages	false		feature::windows_rce	update	/system/default/web-features.conf	2023-09-08
enable_autoformatted_comments	false		feature:search_auto_format	update	/system/default/web-features.conf	2023-09-08
enable_dashboards_external_content_restriction	true		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
enable_dashboards_redirection_restriction	true		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
enable_events_viz	true		feature:dashboard_studio	update	/system/default/web-features.conf	2023-09-08
enable_home_vnext	true		feature:page_migration	update	/system/default/web-features.conf	2023-09-08
enable_inputs_on_canvas	true		feature:dashboard_studio	update	/system/default/web-features.conf	2023-09-08
enable_jQuery2	false	true	feature:quarantine_files	update	/system/default/web-features.conf	2023-09-08
enable_search_v2_endpoint	false		feature:search_v2_endpoint	update	/system/default/web-features.conf	2023-09-08
enable_share_job_control	true		feature:share_job	update	/system/default/web-features.conf	2023-09-08
enable_show_hide	true		feature:dashboard_studio	update	/system/default/web-features.conf	2023-09-08
enable_triggered_alerts_vnext	true		feature:page_migration	update	/system/default/web-features.conf	2023-09-08
enable_unsupported_hotlinked_imports	false	true	feature:quarantine_files	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.flowmilldocs	docs.flowmill.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.rigorhelp	help.rigor.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.splunkapps	apps.splunk.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.splunkbase	splunkbase.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.splunkbasesplunk	splunkbase.splunk.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.splunkdev	dev.splunk.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.splunkdocs	docs.splunk.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.splunkdownload	www.splunk.com/download		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.splunklantern	lantern.splunk.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.splunkproducts	www.splunk.com/products		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.splunkui	splunkui.splunk.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08
internal.dashboards_trusted_domain.victoropshelp	help.victorops.com		feature:dashboards_csp	update	/system/default/web-features.conf	2023-09-08

I see that you just upgraded from 8 to 9 so this new index doesn't contain data from your upgrade.  But you can manually ingest $SPLUNK_HOME/var/log/splunkd/migration.log.* as a one-time job, then do a similar search

source=*/splunk/migration.log.* (data.path=*/system/default/web-features.conf)
| spath path=data.changes{}
| fields - data.changes{}.*
| mvexpand data.changes{}
| spath input=data.changes{}
| spath input=data.changes{} path=properties{}
| fields - properties{}.*
| mvexpand properties{}
| spath input=properties{}
| stats latest(*_value) as *_value by data.action name stanza data.path _time
| eval data.path = replace('data.path', ".*/[sS]plunk/etc", "")
| fieldformat _time = strftime(_time, "%F")
| table name *_value stanza data.* _time

If your system has lots of history, the output will be numerous.  You should probably limit search to the last archived migration.log. (The current one is created after your upgrade, so historic data is not in.)

Hope this helps

[ymb]

Free license does come with few limitations, which are outlined in below doc: https://docs.splunk.com/Documentation/Splunk/latest/Admin/MoreaboutSplunkFree

I was able to reproduce the same behavior when I switched my instance to use a Free license and started getting:

Unable to load common tasks. Refresh the page to try again

and the app list wouldn't load in the UI. 

I switched back to an Enterprise license and that got rid of the UI errors.