Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

[brent_weaver]

For some reason I am unable to do searches behind my Azure load balancer, although it once worked. When I inspect the element on the web page I get the following:

https://logsearch.domain.com:8000/en-US/splunkd/__raw/servicesNS/admin/search/search/jobs Failed to load resource: the server responded with a status of 401 (Splunk cannot authenticate the request. CSRF validation failed.)

Does anyone have any thoughts? Perhaps a DNS issue?

Thanks!

[vince2010091]

Form my side, that was the httponly flag added by the reverse proxy on cookies

[brent_weaver · Accepted solution]

I resolved this issue by clearing my cache in my chrome browser.

Thank you ALL for taking the time to help me out here.

[brent_weaver]

It is important to note in this issue that this problem only exists when referring to the load balancer, if I go to each individual node it works just fine. It does give a warning because the SSL cert we are using is registered as the hostname of the loadbalancer and not the individual node. i dont think that splunk is "broken" per se, i think it is perhaps a configuration that needs to be done?!?!

As always, thank you all for taking the time to help me out here.

[jkat54]

Good clarification, thanks!

[Lucas_K]

Any particular reason you are trying to do an API call against the web interface instead of the management port? (8089 by default?)

Is your load balancer port 8000 redirecting to 8089?

[jkat54]

He's not browsing his API on 8089... he's setup SSL so https:///host:8000 is good.

[brent_weaver]

I have done nothing out of the ordinary. Nothing is specifically configured, how can you tell that I am looking to port 8089. Any further help is MUCH appreciated!

[jplumsdaine22]

I don't know how your load balancer is configured, but I would guess it isn't handling the client session correctly. CRSF sounds like a session issue. Can you connect directly to a searchhead? Do you get the same problem? If you don't have any errors connecting directly to the search head then you have a problem between the lb and the search head. Could be a few different things but at least it will rule out problems with Splunk itself

[jkat54]

Looks like your SSL cert has expired or is otherwise experiencing difficulties.

[brent_weaver]

Thank you VERY much for the quick response! What leads you to believe this? We just got the certs and they are set to expire in 3 years!?! Not saying that isnt the issue, just confusing to me. What are some further troubleshooting steps I can take?

Again, thank you VERY much for your quick response!

[jkat54]

Please send us your "web.conf" on your search heads (sensitive info & ssl key password redacted if exists)

The file is in
$SPLUNK_HOME/etc/system/local and $SPLUNK_HOME/etc/system/default usually.

But we would be interested in the debug.txt created by the following debug command instead:
$SPLUNK_HOME/bin/splunk cmd btool web list --debug > debug.txt

(again remove passwords)

Also, please send the VIP configuration on the load balancer, and brand / type of load balancer.

[brent_weaver]

This is what I am getting when I try:

12/15/15
9:15:59.810 AM

12-15-2015 14:15:59.810 +0000 ERROR UiAuth - Request from 172.16.2.11 to "/en-US/splunkd/__raw/servicesNS/admin/search/search/jobs" had multiple CSRF cookies with different values (first "4646275108905813148" then "12739196604488450756"

Should I clear my browser?

[brent_weaver]

Actually I just resolved this by merely clearing my browser cache in chrome!?!?!?

What the heck, i spent weeks on this lol!!

[jplumsdaine22]

lol. Glad you got it fixed in the end! Do you mind making a new answer in this thread and accepting it? So other people that have the same problem will be able to see how you fixed it