I have search head clustering and SSO set up with reverse proxy authentication, but why do my searches produce "Server Error"?

[ishaanshekhar]

I have a search head cluster and I set up SSO with reverse proxy authentication and set scripted authorization using a Python script. This is working very fine. This setup is within intranet.

Later I added another reverse proxy which is exposed to internet and internally route to intranet server. This also works andSplunk homepage does open after successful authentication and authorization.

However, none of my searches are working. For every search, the message says "Server Error".

In short, Splunk searches are working when it is opened using direct Splunk server URL and using intranet URL. But, it none of the searches are working using internet URL.

I checked splunkd.log and found below message:

ERROR UiAuth - Request from <SH_IP_address> to "/splunk/en-US/splunkd/__raw/servicesNS/<user_name>/search/search/search/jobs" failed CSRF validation -- expected "4647222401877220", but instead cookie had "4647222401877220" and header had ""

splunkd_ui_access.log:

SH_IP_address - User_Name [Date Time] "POST /splunk/en-US/splunkd/__raw/servicesNS/User_Name/search/search/jobs HTTP/1.1" 401 104 "https://Internet_URL/en-US/app/search/search?q=search%20index%3D_internal" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_4 like Mac OS X) ...." - 53f012485f2fb9d 0ms

Sample success message in splunkd_ui_access.log from Intranet URL

SH_IP_address - User_Name [Date Time] "POST /splunk/en-US/splunkd/__raw/servicesNS/nobody/search/search/jobs/1454684456.680_EB62A3AA-75CD-40-A71C-DA6DDB53F181/control HTTP/1.1" 200 59 "https://Intranet_URL/en-US/app/search/search?q=search%20index%3D_internal&display.page.search.mode=smart&earliest=&latest=&sid=14546456.680_EB62A3AA-75CD-4600-A71C-DA6DDB53F181" "Mozilla/5.0 (Windows NT 6.1; WOW64) ...." - 1e371eb6 3ms

How should I rectify the problem with internet URL?

Thanks
Ishaan

[davietch]

Hi,

Were you able to fix this?
I've got the exact same issue :

failed CSRF validation -- expected "123456", but instead cookie had "123456" and header had ""

I've got Splunk 7.2.7 behind a Reverse proxy.

[deepashri_123]

Hi,

I am facing the same issue , and i have a single search head. Did you find any solution?

[JHudson_CVX]

Can you share you email Id? I have created detailed documentation in word for Setting up IIS as a reverse proxy with Splunk. I will email you...Thanks...

[mukuldang08]

CAn you send me the same? mukul94dang@gmail.com

[deepashri_123]

hey@JHudson_CVX,
Appreciate ur help!!!
my email id is: deepashri.amrutkar@smartcirqls.com

[JHudson_CVX]

I sent you an email. Hope you got it...

[vidyadharms]

We have a similar setup and we are using IIS 8.5 as Reverse Proxy....But only redirection happens and SSO didn't work. We got see only Splunk login page. No value for X-Remote-User variable when checked in Splunk SSO debug page. Can you please share the steps to configure SSO with IIS reverse proxy authentication?

[jplumsdaine22]

Hi @ishaanshekhar

How are you controlling the route persistence from the reverse proxy to the search head cluster? Make sure you follow the guide here (http://docs.splunk.com/Documentation/Splunk/6.3.3/DistSearch/UseSHCwithloadbalancers)

Also how have you configured the load balancer? Are you doing any header / cookie rewriting?

[ishaanshekhar]

Someone please help!!!!

[deepashri_123]

Hi@ishaanshekhar,
Can u help how u resolved this issue