spotlight_search results in "The requested URL was not found on this server." Splunk behind reverse proxy

[dstoev]

Hello all, I have the following case:
Splunk accessible on https://dh2.mydomain.com/sendemail931 with "enable_spotlight_search = true" in web-features.conf. If I search for anything and a result/match is shown upon clicking I get "The requested URL was not found on this server.", because the root_endpoint is being removed from the URL. Splunk is behind a reverse proxy (httpd) and an applicaiton load balancer. So upon clicking on the result, I'm being redirected to: https://dh2.mydomain.com/manager/launcher/admin/alert_actions/email?action=edit, but it should be  https://dh2.mydomain.com/sendemail931/en-US/manager/launcher/admin/alert_actions/email?action=edit

I'm pretty sure that the redirect is happening internally, because I cannot see any relevant logs on the apache.

I've tried to add the following to the web.conf, but the result is the same:

tools.proxy.base = https://dh2.mydomain.com/sendemail931/
tools.proxy.on = true

This is the only case were root_endpoint is not preserved.

I've tried to reverse-engineer why this could happen and found that the request is handled by common.min.js, I guess somewhere here:

{title:(0,r._)("Alert actions"),id:"/servicesNS/nobody/search/data/ui/manager/alert_actions",description:(0,r._)("Review and manage available alert actions"),url:"/manager/".concat(e,"/alert_actions"),keywords:[]}

 + here:

{var o=m.default.getSettingById(r);if(void 0===o)return;return(0,b.default)(o.title,o.url,O.length),n=o.url,void(window.location.href=n)

[dstoev · Accepted solution]

This is the official reply from the Splunk team:

"The Splunk team was reported with the similar issue "search settings" in 'settings' doesn't work - Page not found error" just few months before you reported the issue. Whereas the issue was with a different version and not with the Europium instance. The bug was fixed for the upcoming versions but not for the europium release.

The Splunk team was able to reproduce the issue, but only without the proxy. Due to some limitations, the team was not able to reproduce the issue with respect to the proxy. Hence, the developer required additional details to find the cause for the bug as the developer doesn't have the access to your stack.

The bug was reported and fixed recently. Hence, the issue is not mentioned over any of the release notes. Currently, we do not have any ETA regarding this issue to be documented for affected versions (9.2.x,9.3.x and 9.4.x)."