Splunk Enterprise

Why is Splunk is not showing the menu and the UI as expected?

glpadilla_sol
Path Finder

Hello community,

since a couple of months ago we are having an issue into Splunk and is so weird...

The issue is that we are working as usual and suddenly we lost the session and when we refresh the UI is shown like the image below:

glpadilla_sol_0-1662484348275.png

So as you can see the menu at the right top is broken and also the icon is broken.

Also if you try to click one of the available option, don't work.

We have a cluster search head and a Load Balancer.

 

If you have any idea I'll really appreciate it.

 

Thanks in advance.

Version 8.2.2

Labels (1)
Tags (2)
0 Karma

thesplunkmonkey
Path Finder

Since you're using a cluster, ensure that your sessions are persistent for UI traffic.  If you look in the developer tools in your browser, you may see 404 messages being thrown for various authentication related items.  This occurs when the user has logged in, but subsequent calls by the UI itself are made as the user back to the VIP and get routed to different members in the cluster that don't have the rest of your session info.  Refreshing sometimes will help temporarily in this scenario if all calls happen to hit the same member. 

0 Karma

glpadilla_sol
Path Finder

Hi @thesplunkmonkey  thanks for the answer.

The persistence should be at the load balancer right? 

I think that could be the root cause because the issue starts some time after the Load Balancer was implemented... 

Kind Regards.

0 Karma

thesplunkmonkey
Path Finder

Yes, exactly.  If you are using a global LB to balance between LTMs, you'll want to make sure that your persistence is set properly at both the global and local levels so that not only do you stay pinned to the same data center, but also the same host in that data center. 

0 Karma

glpadilla_sol
Path Finder

Perfect @thesplunkmonkey 

 

Now something very weird happened.

Suddenly Splunk is showing our stanza as Splunk Cloud, when is not a Splunk cloud is Enterprise on premise. 

Look:

glpadilla_sol_0-1662654862427.png

 

I thinks there is something weird happening, but I cannot find the root cause at the internal logs.

0 Karma

diogofgm
SplunkTrust
SplunkTrust

I've seen that happening in some of my on-prem customers. Still no idea why that happens but just refreshing the page fixes it.

------------
Hope I was able to help you. If so, some karma would be appreciated.
0 Karma

isoutamo
SplunkTrust
SplunkTrust

Same happening also for me with Splunk 9.0.1 version. Just show that Splunk Cloud and after I do something on that screen it change back to Enterprise. Definitely some kind of bug, but I cannot reproduce it to create bug report to splunk.

r. Ismo

0 Karma

thesplunkmonkey
Path Finder

Interesting.  I haven't seen this exact logo change (splunk>enterprise to splunk>cloud), but I have seen it change before for other Splunk products, i.e. Splunk> hunk.  I'm not sure, but it could be related to something in your license.  When I've seen it change to the hunk logo, it ended up being due to a stanza in the license file referencing hunk, even though we weren't using it.  In fact, when we saw the hunk change, it was actually due to a demo license file included with the product and we just needed to delete that file.  In this case I would first look in the license files that may exist on your search heads and on the license manager to see if there is any reference to Splunk Cloud on any of those.

All that being said, you may consider opening a new question as this seems like a separate topic from the original post.

0 Karma

glpadilla_sol
Path Finder

Thanks for all the inputs, yes this a different topic and I found this is a known-issue of this version which is weird.

https://docs.splunk.com/Documentation/Splunk/8.2.2/ReleaseNotes/KnownIssues#Distributed_search_and_s...

Because I haven't seen that before.

Anyways I'll be working checking the LB configuration to see what is the root cause of the initial issue I raised.

Thanks again.

0 Karma

thesplunkmonkey
Path Finder

Any time!  Once you have had a chance to implement, and assuming it works for you, consider dropping back in and accepting that solution so others can reference it.    And if not, please let us know that too.

0 Karma

glpadilla_sol
Path Finder

Hello @thesplunkmonkey 

 

We notice that the LB has a sticky session using http cookies, but that session expires every hour and casually the error happens each time the session expires and we have to re-login...

But if we re-logging we get the session and usually with other server.

So I think the issue is when the cookie expires and they have to provide another one that interval causes some issue, because when we refresh the behavior disappears.

Do you have any idea about what should be configure at the LB, I found this in another post - But I am not sure about what that means --- please check "session_cookie_insert' as the default persistence profile". 

https://community.splunk.com/t5/Splunk-Search/Why-am-I-getting-search-error-quot-In-handler-savedsea...

Also the LB admin told me he can configure the session to expires at the end of a browser session but I am not sure if that is going to help of cause another issue.

 

THANKS in advance.

0 Karma

thesplunkmonkey
Path Finder

Yes, use the cookie insert persistence for your LTMs and topology at the GTM.  In that way you'll be routed to the same data center with subsequent calls and then the cookie insert persistence will keep you pinned to the same SH in that site with subsequent calls.   Your LB admin should know how to do the configs for that type of persistence for whatever particular type of LB you're running.

0 Karma

matt8679
Path Finder

Looks like you have a permission issue. Check that your role has the proper read/write to apps.

0 Karma
Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...