Splunk Search

Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

Builder

For some reason I am unable to do searches behind my Azure load balancer, although it once worked. When I inspect the element on the web page I get the following:

https://logsearch.domain.com:8000/en-US/splunkd/__raw/servicesNS/admin/search/search/jobs Failed to load resource: the server responded with a status of 401 (Splunk cannot authenticate the request. CSRF validation failed.)

Does anyone have any thoughts? Perhaps a DNS issue?

Thanks!

1 Solution

Builder

I resolved this issue by clearing my cache in my chrome browser.

Thank you ALL for taking the time to help me out here.

View solution in original post

Path Finder

Form my side, that was the httponly flag added by the reverse proxy on cookies

0 Karma

Builder

I resolved this issue by clearing my cache in my chrome browser.

Thank you ALL for taking the time to help me out here.

View solution in original post

Builder

It is important to note in this issue that this problem only exists when referring to the load balancer, if I go to each individual node it works just fine. It does give a warning because the SSL cert we are using is registered as the hostname of the loadbalancer and not the individual node. i dont think that splunk is "broken" per se, i think it is perhaps a configuration that needs to be done?!?!

As always, thank you all for taking the time to help me out here.

0 Karma

SplunkTrust
SplunkTrust

Good clarification, thanks!

0 Karma

Motivator

Any particular reason you are trying to do an API call against the web interface instead of the management port? (8089 by default?)

Is your load balancer port 8000 redirecting to 8089?

0 Karma

SplunkTrust
SplunkTrust

He's not browsing his API on 8089... he's setup SSL so https:///host:8000 is good.

0 Karma

Builder

I have done nothing out of the ordinary. Nothing is specifically configured, how can you tell that I am looking to port 8089. Any further help is MUCH appreciated!

0 Karma

Influencer

I don't know how your load balancer is configured, but I would guess it isn't handling the client session correctly. CRSF sounds like a session issue. Can you connect directly to a searchhead? Do you get the same problem? If you don't have any errors connecting directly to the search head then you have a problem between the lb and the search head. Could be a few different things but at least it will rule out problems with Splunk itself

0 Karma

SplunkTrust
SplunkTrust

Looks like your SSL cert has expired or is otherwise experiencing difficulties.

0 Karma

Builder

Thank you VERY much for the quick response! What leads you to believe this? We just got the certs and they are set to expire in 3 years!?! Not saying that isnt the issue, just confusing to me. What are some further troubleshooting steps I can take?

Again, thank you VERY much for your quick response!

0 Karma

SplunkTrust
SplunkTrust

Please send us your "web.conf" on your search heads (sensitive info & ssl key password redacted if exists)

The file is in
$SPLUNK_HOME/etc/system/local and $SPLUNK_HOME/etc/system/default usually.

But we would be interested in the debug.txt created by the following debug command instead:
$SPLUNK_HOME/bin/splunk cmd btool web list --debug > debug.txt

(again remove passwords)

Also, please send the VIP configuration on the load balancer, and brand / type of load balancer.

0 Karma

Builder

This is what I am getting when I try:

12/15/15
9:15:59.810 AM

12-15-2015 14:15:59.810 +0000 ERROR UiAuth - Request from 172.16.2.11 to "/en-US/splunkd/__raw/servicesNS/admin/search/search/jobs" had multiple CSRF cookies with different values (first "4646275108905813148" then "12739196604488450756"

Should I clear my browser?

0 Karma

Builder

Actually I just resolved this by merely clearing my browser cache in chrome!?!?!?

What the heck, i spent weeks on this lol!!

0 Karma

Influencer

lol. Glad you got it fixed in the end! Do you mind making a new answer in this thread and accepting it? So other people that have the same problem will be able to see how you fixed it

0 Karma