after upgrading to 8.0.2 from 7.3.1, splunkweb won't start. after I remove the search activity app it starts again.
This may be because the app packages a "custom module" in the legacy Advanced XML system, and that the custom module contains its own custom python endpoint.
You can check it very quickly by looking for a stack trace in web_service.log
If there is one, look carefully in the last line of the stack - it will refer to a python file that is in a particular app, in an "appserver/modules" subdirectory.
If that stack trace does exist, and points to this particular app, that's your root cause.
And if so, here's the explanation of what just happened>
a) those endpoints are not a part of the normal "web.conf" endpoints that you might be more familiar with in Splunkweb. This stuff is a pretty obscure backwater of some ancient systems that predate the web.conf controllers and have been largely forgotten since 4.X/5/X days.
b) Splunkd in 8.0 still runs those legacy "advanced xml custom module" endpoints even though technically the advanced XML was suposed to be removed completely in 8.0. Splunk has acknowledged this as a mistake and I hear it's being fixed.
c) furthermore Splunkd in 8.0 runs these ancient things in python3, which will almost guarantee that the endpoint wont run.
d) more fun - when it fails to run the endpoint, splunkweb then itself fails to start for some reason.
Is it just splunkweb that will not start or both splunkweb and splunkd?