Installation

Why does Splunk assist constantly fail on 9.1.0.1 version after upgrade due to free license?

ori
New Member

hey, we're in the process of upgrading on our splunk single instances from 8.2.5 to 9.1.0.1 due to EOL.

we have two splunk instances - production which is under the enterprise license, test which is under the free license.

we've first started upgrading our test splunk instance with the free license and we instantly saw these errors:

 

 

07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py" [modular_input:349] [execute] [834704] Modular input: Splunk Assist exit with exception: Traceback (most recent call last):
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/assist/modular_input.py", line 342, in execute
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     self.do_run(input_definition["inputs"])
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py", line 66, in do_run
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     if not should_run(self.logger, self.session_key):
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py", line 27, in should_run
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     sh = is_search_head(log, session_key)
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/assist/serverinfo.py", line 153, in is_search_head
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     cluster_mode = get_cluster_mode(log, session_key)
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/etc/apps/splunk_assist/bin/assist/serverinfo.py", line 257, in get_cluster_mode
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     raiseAllErrors=True
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"   File "/opt/splunk/lib/python3.7/site-packages/splunk/rest/__init__.py", line 646, in simpleRequest
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py"     raise splunk.LicenseRestriction
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py" splunk.LicenseRestriction: [HTTP 402] Current license does not allow the requested action
07-31-2023 07:01:44.412 +0000 ERROR ExecProcessor [670742 ExecProcessor] - message from "/opt/splunk/bin/python3.7 /opt/splunk/etc/apps/splunk_assist/bin/uiassets_modular_input.py" .

 

 

 

this splunk assist issue seems to come from the usage of free license on splunk.

I tried to disable the splunk-assist app but it wouldn't let me:

 

 

Cannot disable app: splunk_assist

 

 

 
as a result of the previous errors, we are seeing UI errors as well: 

 

 

Unable to load common tasks. Refresh the page to try again.

 

 

 

any idea on how to proceed?

Labels (1)
Tags (2)
0 Karma

yuanliu
SplunkTrust
SplunkTrust

Free license or not free license, Splunk UI should not display cryptic error message.  I consider this a usability bug because the license type can easily be handled with clearer message.

You can disable Splunk Assist.  But that will not solve this problem.

To disable the application, go to $SPLUNK_HOME/etc/apps/splunk_assist.  Create a directory local/, then create a file app.conf to override allow_disable=true in [install] stanza, like this:

 

cd $SPLUNK_HOME/etc/apps/splunk_assist
mkdir local
cat <<EOM >local/app.conf
[install]
allows_disable = false
EOM

 

 

 

After restarting Splunk, you can then disable splunk_assist.  But the error will persist even when Splunk Assist is disabled.  BTW, if you launch splunk_assist directly, you will be able to see the error message

Unable to obtain template "beam:/templates/start.html": ... TopLevelLookupException(_("Splunk has failed to locate the template for uri '%s'." % uri)) TopLevelLookupException: Splunk has failed to locate the template for uri 'beam:/templates/start.html'.

The problem is that a 9.1 upgrade enables a feature called enable_home_vnext.  It is a fine feature (with tons of marketing plugs) except for this bug: Splunk should detect the free license and handle the error without showing a cryptic error message. (It can show, for example, "this feature is unavailable for the installed license" like it does with a bunch of other features.)

Workaround

The only workaround I have discovered so far is to disable enable_home_vnext in [feature:page_migration] stanza from web-features.conf.  ([feature:page_migration] is introduced in 9.x but is in web.conf. web-features.conf is introduced in 9.1.1; a skeleton local/web-features.conf is created by installer but the stanza is not in it.)

 

 

cd $SPLUNK_HOME/etc/system/local
cat <<EOM>>web-features.conf
[feature:page_migration]
enable_home_vnext = false
EOM

 

 

After this, you will see your previous launcher home page instead of the migration page. (You may need to to restart splunkd.  I had at least one instance for which I did not perform restart after editing.)  I hope Splunk will not take the legacy launch page away before fixing the bug in the new page.

What changed?

The new index _configtracker added in 9.x makes it much easier to track changes made over time.  Because I was upgrading from 9.0.5, I can see the exact changes 9.1 upgrade has made.  For example, to see web feature changes,

 

 

index="_configtracker" (data.path=*/system/default/web-features.conf)
| spath path=data.changes{}
| fields - data.changes{}.*
| mvexpand data.changes{}
| spath input=data.changes{}
| spath input=data.changes{} path=properties{}
| fields - properties{}.*
| mvexpand properties{}
| spath input=properties{}
| stats latest(*_value) as *_value by data.action name stanza data.path _time
| eval data.path = replace('data.path', ".*/[sS]plunk/etc", "")
| fieldformat _time = strftime(_time, "%F")
| table name *_value stanza data.* _time

 

 

On my laptop instance (defaults), this shows

namenew_valueold_valuestanzadata.actiondata.path_time
disable_highcharts_accessibilityfalse feature:highcharts_accessibilityupdate/system/default/web-features.conf2023-09-08
enable_acuif_pagesfalse feature::windows_rceupdate/system/default/web-features.conf2023-09-08
enable_autoformatted_commentsfalse feature:search_auto_formatupdate/system/default/web-features.conf2023-09-08
enable_dashboards_external_content_restrictiontrue feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
enable_dashboards_redirection_restrictiontrue feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
enable_events_viztrue feature:dashboard_studioupdate/system/default/web-features.conf2023-09-08
enable_home_vnexttrue feature:page_migrationupdate/system/default/web-features.conf2023-09-08
enable_inputs_on_canvastrue feature:dashboard_studioupdate/system/default/web-features.conf2023-09-08
enable_jQuery2falsetruefeature:quarantine_filesupdate/system/default/web-features.conf2023-09-08
enable_search_v2_endpointfalse feature:search_v2_endpointupdate/system/default/web-features.conf2023-09-08
enable_share_job_controltrue feature:share_jobupdate/system/default/web-features.conf2023-09-08
enable_show_hidetrue feature:dashboard_studioupdate/system/default/web-features.conf2023-09-08
enable_triggered_alerts_vnexttrue feature:page_migrationupdate/system/default/web-features.conf2023-09-08
enable_unsupported_hotlinked_importsfalsetruefeature:quarantine_filesupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.flowmilldocsdocs.flowmill.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.rigorhelphelp.rigor.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.splunkappsapps.splunk.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.splunkbasesplunkbase.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.splunkbasesplunksplunkbase.splunk.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.splunkdevdev.splunk.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.splunkdocsdocs.splunk.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.splunkdownloadwww.splunk.com/download feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.splunklanternlantern.splunk.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.splunkproductswww.splunk.com/products feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.splunkuisplunkui.splunk.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08
internal.dashboards_trusted_domain.victoropshelphelp.victorops.com feature:dashboards_cspupdate/system/default/web-features.conf2023-09-08

I see that you just upgraded from 8 to 9 so this new index doesn't contain data from your upgrade.  But you can manually ingest $SPLUNK_HOME/var/log/splunkd/migration.log.* as a one-time job, then do a similar search

 

 

source=*/splunk/migration.log.* (data.path=*/system/default/web-features.conf)
| spath path=data.changes{}
| fields - data.changes{}.*
| mvexpand data.changes{}
| spath input=data.changes{}
| spath input=data.changes{} path=properties{}
| fields - properties{}.*
| mvexpand properties{}
| spath input=properties{}
| stats latest(*_value) as *_value by data.action name stanza data.path _time
| eval data.path = replace('data.path', ".*/[sS]plunk/etc", "")
| fieldformat _time = strftime(_time, "%F")
| table name *_value stanza data.* _time

 

 

If your system has lots of history, the output will be numerous.  You should probably limit search to the last archived migration.log. (The current one is created after your upgrade, so historic data is not in.)

Hope this helps

Tags (1)
0 Karma

ymb
Splunk Employee
Splunk Employee

Free license does come with few limitations, which are outlined in below doc: https://docs.splunk.com/Documentation/Splunk/latest/Admin/MoreaboutSplunkFree

I was able to reproduce the same behavior when I switched my instance to use a Free license and started getting:

Unable to load common tasks. Refresh the page to try again

and the app list wouldn't load in the UI. 

I switched back to an Enterprise license and that got rid of the UI errors.

Get Updates on the Splunk Community!

2024 Splunk Career Impact Survey | Earn a $20 gift card for participating!

Hear ye, hear ye! The time has come again for Splunk's annual Career Impact Survey!  We need your help by ...

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...