Why is Splunk is not showing the menu and the UI as expected?

[glpadilla_sol]

Hello community,

since a couple of months ago we are having an issue into Splunk and is so weird...

The issue is that we are working as usual and suddenly we lost the session and when we refresh the UI is shown like the image below:

So as you can see the menu at the right top is broken and also the icon is broken.

Also if you try to click one of the available option, don't work.

We have a cluster search head and a Load Balancer.

If you have any idea I'll really appreciate it.

Thanks in advance.

Version 8.2.2

[HenderSlender11]

I'm also having a similar problem. The "user menu" for my Splunk UI is simply not there. With this being the case, I'm not able to change my preferences or simply logout. Any help would be greatly appreciated. 

[thesplunkmonkey]

Since you're using a cluster, ensure that your sessions are persistent for UI traffic.  If you look in the developer tools in your browser, you may see 404 messages being thrown for various authentication related items.  This occurs when the user has logged in, but subsequent calls by the UI itself are made as the user back to the VIP and get routed to different members in the cluster that don't have the rest of your session info.  Refreshing sometimes will help temporarily in this scenario if all calls happen to hit the same member. 

[glpadilla_sol]

Hi @thesplunkmonkey  thanks for the answer.

The persistence should be at the load balancer right? 

I think that could be the root cause because the issue starts some time after the Load Balancer was implemented... 

Kind Regards.

[thesplunkmonkey]

Yes, exactly.  If you are using a global LB to balance between LTMs, you'll want to make sure that your persistence is set properly at both the global and local levels so that not only do you stay pinned to the same data center, but also the same host in that data center. 

[glpadilla_sol]

Perfect @thesplunkmonkey 

Now something very weird happened.

Suddenly Splunk is showing our stanza as Splunk Cloud, when is not a Splunk cloud is Enterprise on premise. 

Look:

I thinks there is something weird happening, but I cannot find the root cause at the internal logs.

[diogofgm]

I've seen that happening in some of my on-prem customers. Still no idea why that happens but just refreshing the page fixes it.

[isoutamo]

Same happening also for me with Splunk 9.0.1 version. Just show that Splunk Cloud and after I do something on that screen it change back to Enterprise. Definitely some kind of bug, but I cannot reproduce it to create bug report to splunk.

r. Ismo

[thesplunkmonkey]

Interesting.  I haven't seen this exact logo change (splunk>enterprise to splunk>cloud), but I have seen it change before for other Splunk products, i.e. Splunk> hunk.  I'm not sure, but it could be related to something in your license.  When I've seen it change to the hunk logo, it ended up being due to a stanza in the license file referencing hunk, even though we weren't using it.  In fact, when we saw the hunk change, it was actually due to a demo license file included with the product and we just needed to delete that file.  In this case I would first look in the license files that may exist on your search heads and on the license manager to see if there is any reference to Splunk Cloud on any of those.

All that being said, you may consider opening a new question as this seems like a separate topic from the original post.

[glpadilla_sol]

Thanks for all the inputs, yes this a different topic and I found this is a known-issue of this version which is weird.

https://docs.splunk.com/Documentation/Splunk/8.2.2/ReleaseNotes/KnownIssues#Distributed_search_and_s...

Because I haven't seen that before.

Anyways I'll be working checking the LB configuration to see what is the root cause of the initial issue I raised.

Thanks again.

[thesplunkmonkey]

Any time!  Once you have had a chance to implement, and assuming it works for you, consider dropping back in and accepting that solution so others can reference it.    And if not, please let us know that too.

[glpadilla_sol]

Hello @thesplunkmonkey 

We notice that the LB has a sticky session using http cookies, but that session expires every hour and casually the error happens each time the session expires and we have to re-login...

But if we re-logging we get the session and usually with other server.

So I think the issue is when the cookie expires and they have to provide another one that interval causes some issue, because when we refresh the behavior disappears.

Do you have any idea about what should be configure at the LB, I found this in another post - But I am not sure about what that means --- please check "session_cookie_insert' as the default persistence profile". 

https://community.splunk.com/t5/Splunk-Search/Why-am-I-getting-search-error-quot-In-handler-savedsea...

Also the LB admin told me he can configure the session to expires at the end of a browser session but I am not sure if that is going to help of cause another issue.

THANKS in advance.

[thesplunkmonkey]

Yes, use the cookie insert persistence for your LTMs and topology at the GTM.  In that way you'll be routed to the same data center with subsequent calls and then the cookie insert persistence will keep you pinned to the same SH in that site with subsequent calls.   Your LB admin should know how to do the configs for that type of persistence for whatever particular type of LB you're running.

[matt8679]

Looks like you have a permission issue. Check that your role has the proper read/write to apps.