Problem with DEPLOYMENT after downgrading from 9.4.x to 9.1.9

[verbal_666]

Hi there.

I got a really hard problem after updating to 9.1.9.
DS completely broke, can't manage it anymore.

Is there a fast solution?
Thanks.

[verbal_666]

OK, fixed 😇

The problem was a system addon ("search", IMO), maybe previously placed under $SPLUNK_HOME/etc/apps/ which breaks totally the UI for the DS.

I simply deleted all system addons in $SPLUNK_HOME/etc/apps/ and untar again the main 9.1.9 Enterprise package. So all apps are restored to their default and do not contain maybe a next update (9.4.x for example).

The problem was a rollback from 9.4.x to 9.1.9 . I usually maintain all my "etc", but it's hard to think something inside the system addon could break something 🙄

[verbal_666]

Yes. Addon is the "search" one.

If someone wants to roolback 9.4.x to 9.1/9.2/9.3/... needs,

- stop instance
- delete/rename the $SPLUNK_HOME/etc/apps/search/default
- untar the 9.1/9.2/9.3/... again
- restart instance

In any other way the new 9.4.x "search/default" makes the Deployment UI KO 😲

[PickleRick]

Just to be clear. There is no supported way of downgrading Splunk installation. So your initial problem was not caused by _upgrade_ directly from 8.2 to 9.1 but by your attempt to downgrade from 9.4 to which you had already upgraded mid-flight. And your "solution" while it might have worked in your particular case is not something that should be considered a safe solution.

First and foremost - you shouldn't downgrade. As simple as that.

[verbal_666]

No.

The problem was by,

1. updating 8.2.x to 9.1.9 [OK]
2. updating 9.1.9 to 9.4.4 [KO... KVSTORE stops starting... and updating from V4>>V7]

So i remained stuck at 9.4.4 with KVSTORE not working in SHCluster!!! And that's a great problem!!!

So i need to,

3. downgrading from 9.4.4 to 9.1.9

And now the /apps/search/default totally broke che DS UI 😳 IMO it's not normal!!!

[PickleRick]

There are some issues with kvstore upgrade, typically caused by SSL-related issues, yes. They are more or less well docummented. But they have nothing to do with deployment server or its "breaking".

The proper upgrade procedure should include (and I think it is explicitly stated in the docs) backup prior to upgrade. Should the upgrade fail you're supposed to restore the environment from backup to last working state, not do downgrades from an unknown state using an unsupported untested and unreliable methods.

[verbal_666]

I always do a full backup before updating,

tar -zcf /my_path_for_backups/splunk_actual.tgz /opt/splunk

I don't understand why it bothers you so much if i deal with cases where any user could end up, providing documentation that could be useful!!!

Since i love I.T. , and Splunk, i really like to enter many cases and seek for a clue to remediate issues.

Does it bother you? Byebye.

PS. reading around, i found others users with this DS issue!!!

[PickleRick]

It's not that I'm bothered by what you did as such.

It's just that I don't think that promoting unsupported actions is healthy. And I think that if someone finds this thread they should have the full picture.

So while I fully sympathize and understand why you did what you did, the reader deserves a full explanation (and the title of this thread is misleading because the upgrade from 8.2 to 9.1 did _not_ cause any problems on its own).

You were trying to salvage a situation you put yourself into. What you shouid have done after upgrade from 9.1 to 9.4 was to either resolve the problem with kvstore (which I admit could have been problematic and non obvious) or - if you weren't able to fix the kvstore - restore from backup.

"Downgrading" was the worst thing you could have done because there are things which change between releases (otherwise there would be no need for upgrade) which will not work with earlier releases. At this point you "only" ended up with not working DS gui but you might have ended up with broken indexes, or kvstore if it was an upgrade introducing changes to internal data format.

So while in your particular situation this worked, it is not a general solution to the problem of broken upgrade and it should not be promoted as such.

Especially if others have encountered similar problems after properly performed upgrade. Your "solution" which is only a walkaround for a problem you created yourself, is not helping others.

I'm glad that it worked for you. It might be an interesting piece of information but with the proper context.

1. Your problem was not caused by the upgrade from 8 to 9.1 (as the topic says)

2. Your problem was caused (or at least related to) by an unsupported downgrade.

That's it.

Again - I'm glad it worked for you but it's not something that should be marked as solution to "broken DS gui after upgrade from 8.2 to 9.1".

I hope I'm more clear this time.

[verbal_666]

Really do not know, i'm getting mad!!!
No errors, GUI is working perfecly, only the Agent_Management is gone (500 also with old version!!!).
It worked since this morning when i passed it from 8.2.x to 9.4.4 😶

This is a real great problem.

[verbal_666]

2025-10-24 13:14:25,227 INFO    [68fb5f91357fad2682bd10] startup:139 - Splunk appserver version=8.2.12 build=e973afd6886e isFree=False isTrial=False
2025-10-24 13:14:25,290 ERROR   [68fb5f91357fad2682bd10] __init__:351 - Unable to obtain template "pages/page.html":

Traceback (most recent call last):
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/__init__.py", line 346, in render_template
    templateInstance = mako_lookup.get_template(template_name)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/__init__.py", line 227, in get_template
    raise_top_level_lookup_exception(uri)
  File "/opt/splunk/lib/python3.7/site-packages/splunk/appserver/mrsparkle/controllers/__init__.py", line 165, in raise_top_level_lookup_exception
    raise exceptions.TopLevelLookupException(_("Splunk has failed to locate the template for uri '%s'." % uri))
TopLevelLookupException: Splunk has failed to locate the template for uri 'pages/page.html'.

I untar the full 8.2.x package 🙄

What the h3ll is going on???????????????????????????????

[verbal_666]

It's all so illogical.
I rolled back to previous 8.2.x version (i have a full backup).
Now when i enter the UF Management,

It's very frustrating 😳

[PickleRick]

"There is an error in your serverclass.conf".

Did you try 

splunk btool check

? 

[verbal_666]

Yes.
No errors.
Serverclass  config is the same as in 8.2.x 😯

[PickleRick]

That is interesting. What I would try before raising a support case:

1) Verify if the DS itself is working (not just the GUI) - check _internal for phonehomes, check on a sample of forwarders if they're getting their apps correctly.

2) Check _internal for this server's logs and try to find something interesting (yes, I know it's a very vague statement but I don't have anything specific on my mind - it's just a "seek and destroy" operation).

3) If the DS as a whole is not working, I'd try blocking the port so that the clients don't hit it with requests and try to start it with a blank serverclass.conf.

4) Just to be on the safe side - you didn't mess up file permissions?