Splunk Search
Highlighted

Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

Builder

For some reason I am unable to do searches behind my Azure load balancer, although it once worked. When I inspect the element on the web page I get the following:

https://logsearch.domain.com:8000/en-US/splunkd/__raw/servicesNS/admin/search/search/jobs Failed to load resource: the server responded with a status of 401 (Splunk cannot authenticate the request. CSRF validation failed.)

Does anyone have any thoughts? Perhaps a DNS issue?

Thanks!

Highlighted

Re: Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

SplunkTrust
SplunkTrust

Looks like your SSL cert has expired or is otherwise experiencing difficulties.

0 Karma
Highlighted

Re: Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

Builder

Thank you VERY much for the quick response! What leads you to believe this? We just got the certs and they are set to expire in 3 years!?! Not saying that isnt the issue, just confusing to me. What are some further troubleshooting steps I can take?

Again, thank you VERY much for your quick response!

0 Karma
Highlighted

Re: Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

SplunkTrust
SplunkTrust

Please send us your "web.conf" on your search heads (sensitive info & ssl key password redacted if exists)

The file is in
$SPLUNKHOME/etc/system/local and $SPLUNKHOME/etc/system/default usually.

But we would be interested in the debug.txt created by the following debug command instead:
$SPLUNK_HOME/bin/splunk cmd btool web list --debug > debug.txt

(again remove passwords)

Also, please send the VIP configuration on the load balancer, and brand / type of load balancer.

0 Karma
Highlighted

Re: Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

Builder

This is what I am getting when I try:

12/15/15
9:15:59.810 AM

12-15-2015 14:15:59.810 +0000 ERROR UiAuth - Request from 172.16.2.11 to "/en-US/splunkd/__raw/servicesNS/admin/search/search/jobs" had multiple CSRF cookies with different values (first "4646275108905813148" then "12739196604488450756"

Should I clear my browser?

0 Karma
Highlighted

Re: Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

Builder

Actually I just resolved this by merely clearing my browser cache in chrome!?!?!?

What the heck, i spent weeks on this lol!!

0 Karma
Highlighted

Re: Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

Influencer

lol. Glad you got it fixed in the end! Do you mind making a new answer in this thread and accepting it? So other people that have the same problem will be able to see how you fixed it

0 Karma
Highlighted

Re: Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

Influencer

I don't know how your load balancer is configured, but I would guess it isn't handling the client session correctly. CRSF sounds like a session issue. Can you connect directly to a searchhead? Do you get the same problem? If you don't have any errors connecting directly to the search head then you have a problem between the lb and the search head. Could be a few different things but at least it will rule out problems with Splunk itself

0 Karma
Highlighted

Re: Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

Motivator

Any particular reason you are trying to do an API call against the web interface instead of the management port? (8089 by default?)

Is your load balancer port 8000 redirecting to 8089?

0 Karma
Highlighted

Re: Why am I unable to perform searches on a Splunk search head cluster behind an Azure load balancer with error "CSRF validation failed"?

Builder

I have done nothing out of the ordinary. Nothing is specifically configured, how can you tell that I am looking to port 8089. Any further help is MUCH appreciated!

0 Karma
Speak Up for Splunk Careers!

We want to better understand the impact Splunk experience and expertise has has on individuals' careers, and help highlight the growing demand for Splunk skills.