Splunk Search

Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

brent_weaver
Builder

For some reason I am unable to do searches behind my Azure load balancer, although it once worked. When I inspect the element on the web page I get the following:

https://logsearch.domain.com:8000/en-US/splunkd/__raw/servicesNS/admin/search/search/jobs Failed to load resource: the server responded with a status of 401 (Splunk cannot authenticate the request. CSRF validation failed.)

Does anyone have any thoughts? Perhaps a DNS issue?

Thanks!

1 Solution

brent_weaver
Builder

I resolved this issue by clearing my cache in my chrome browser.

Thank you ALL for taking the time to help me out here.

View solution in original post

vince2010091
Path Finder

Form my side, that was the httponly flag added by the reverse proxy on cookies

0 Karma

brent_weaver
Builder

I resolved this issue by clearing my cache in my chrome browser.

Thank you ALL for taking the time to help me out here.

brent_weaver
Builder

It is important to note in this issue that this problem only exists when referring to the load balancer, if I go to each individual node it works just fine. It does give a warning because the SSL cert we are using is registered as the hostname of the loadbalancer and not the individual node. i dont think that splunk is "broken" per se, i think it is perhaps a configuration that needs to be done?!?!

As always, thank you all for taking the time to help me out here.

0 Karma

jkat54
SplunkTrust
SplunkTrust

Good clarification, thanks!

0 Karma

Lucas_K
Motivator

Any particular reason you are trying to do an API call against the web interface instead of the management port? (8089 by default?)

Is your load balancer port 8000 redirecting to 8089?

0 Karma

jkat54
SplunkTrust
SplunkTrust

He's not browsing his API on 8089... he's setup SSL so https:///host:8000 is good.

0 Karma

brent_weaver
Builder

I have done nothing out of the ordinary. Nothing is specifically configured, how can you tell that I am looking to port 8089. Any further help is MUCH appreciated!

0 Karma

jplumsdaine22
Influencer

I don't know how your load balancer is configured, but I would guess it isn't handling the client session correctly. CRSF sounds like a session issue. Can you connect directly to a searchhead? Do you get the same problem? If you don't have any errors connecting directly to the search head then you have a problem between the lb and the search head. Could be a few different things but at least it will rule out problems with Splunk itself

0 Karma

jkat54
SplunkTrust
SplunkTrust

Looks like your SSL cert has expired or is otherwise experiencing difficulties.

0 Karma

brent_weaver
Builder

Thank you VERY much for the quick response! What leads you to believe this? We just got the certs and they are set to expire in 3 years!?! Not saying that isnt the issue, just confusing to me. What are some further troubleshooting steps I can take?

Again, thank you VERY much for your quick response!

0 Karma

jkat54
SplunkTrust
SplunkTrust

Please send us your "web.conf" on your search heads (sensitive info & ssl key password redacted if exists)

The file is in
$SPLUNK_HOME/etc/system/local and $SPLUNK_HOME/etc/system/default usually.

But we would be interested in the debug.txt created by the following debug command instead:
$SPLUNK_HOME/bin/splunk cmd btool web list --debug > debug.txt

(again remove passwords)

Also, please send the VIP configuration on the load balancer, and brand / type of load balancer.

0 Karma

brent_weaver
Builder

This is what I am getting when I try:

12/15/15
9:15:59.810 AM

12-15-2015 14:15:59.810 +0000 ERROR UiAuth - Request from 172.16.2.11 to "/en-US/splunkd/__raw/servicesNS/admin/search/search/jobs" had multiple CSRF cookies with different values (first "4646275108905813148" then "12739196604488450756"

Should I clear my browser?

0 Karma

brent_weaver
Builder

Actually I just resolved this by merely clearing my browser cache in chrome!?!?!?

What the heck, i spent weeks on this lol!!

0 Karma

jplumsdaine22
Influencer

lol. Glad you got it fixed in the end! Do you mind making a new answer in this thread and accepting it? So other people that have the same problem will be able to see how you fixed it

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...