Anyway to see X-Forwarded-For in Splunk access logs?

[jimcroft]

I'm looking to implement some brute force login attempt detection/mitigation for a Splunk cluster. Splunkweb in this case is running behind a loadbalancer so I can't use the source IP from the access logs. I need to rely on the X-Forwarded-For header in the HTTP request but that's not being logged.

Does anyone know how to make splunkweb log the X-Forwarded-For in its access logs?

Thanks

Jim

[maraman_splunk]

Hi,

has somebody made this work with 6.5 or 6.6 ?
I don't see any effect after modifying this file so I'm not sure it's still used by these versions in default webserver configuration ...

[m1k34Splunk]

Just for reference, I've successfully tested the andrewcg solution in 6.6.5.
P.S. Watch out for the correct identation or it won't work.

[john_thom]

Hey andrewcg or m1k34Splunk,
Did you make this change on the search heads themselves? I did it on my search heads and it doesn't appear to be working. I tried including it in atoms['h'] and the access_log write itself (I even tried injecting it into both spots once). Restarted splunkweb after each change but neither seems to have any effect. Still shows the VIP from my load balancer. I'm also on 6.6.x. I feel like I must be missing something silly here.
Thanks!

[andrewcg]

So it looks like the problem with this hack is that it only fixes the logging in:

• web_access.log

The other logs are all unchanged:

• splunkd_ui_access.log
• splunkd_access.log

For example:

splunkd_ui_access.log - shows the proxy

var/log/splunk/splunkd_ui_access.log:127.0.0.1 - admin [13/Feb/2018:11:00:47.110 -0500] "GET /en-US/ HTTP/1.1" 303 105 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - c0027a96a8de667c7653660d00411b6e 18ms
var/log/splunk/splunkd_ui_access.log:127.0.0.1 - admin [13/Feb/2018:11:00:47.183 -0500] "GET /en-US/app/launcher HTTP/1.1" 303 110 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - c0027a96a8de667c7653660d00411b6e 181ms
var/log/splunk/splunkd_ui_access.log:127.0.0.1 - admin [13/Feb/2018:11:00:47.516 -0500] "GET /en-US/app/launcher/home HTTP/1.1" 200 1264 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - c0027a96a8de667c7653660d00411b6e 161ms

web_access.log - shows the X-Forwarded-For

var/log/splunk/web_access.log:10.32.136.60 - admin [13/Feb/2018:11:00:47.111 -0500] "GET /en-US/ HTTP/1.1" 303 105 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - 5a830baf1c7ff401228990 17ms
var/log/splunk/web_access.log:10.32.136.60 - admin [13/Feb/2018:11:00:47.183 -0500] "GET /en-US/app/launcher HTTP/1.1" 303 110 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - 5a830baf2f7ff401228a10 180ms
var/log/splunk/web_access.log:10.32.136.60 - admin [13/Feb/2018:11:00:47.517 -0500] "GET /en-US/app/launcher/home HTTP/1.1" 200 1264 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - 5a830baf847ff4012285d0 159ms

[john_thom]

Ahh, thank you! This is very helpful. I was using the following search which is only showing the LB IP address:
index=_audit action="login attempt" info=failed

When I look at the web_access logs I do indeed see the correct information now:
index=_internal source="*web_access.log"

I'll see if I can poke around today and figure out if there is any way to modify the login attempt entries to check for an XFF header first and if I find anything I'll share with the community.

Thanks for the help, andrewcg!

[hello007]

The log of login attempt action comes from $SPLUNK_HOME/bin/splunkd, there is no other way to modify it for showing the XFF msg.

The web_access.log only log the requests which after login, not the fail login attempt.

[gjanders]

I tried two or three different files that mentioned access_log/atoms and I was unable to get this working...

In fact I couldn't even get the log to update to write the new log statement I had added so I'm unsure which file controls the access log in the newer 6.5.x versions.

[mtulett_splunk]

For any visitors finding this in 2018: The solution from andrewcg's comment above is still working for me in 7.0.

[gjanders]

Thanks for the confirmation, I might try again in Splunk 7...

[martin_mueller · Accepted solution]

Yeah... just not a very nice one.

In $SPLUNK_HOME/Python-2.7/Lib/site-packages/splunk/appserver/mrsparkle/lib/customlogmanager.py's access() method you can fiddle with the log format as you please.
To verify this I've quickly appended line 63 to get this:

self.access_log.log(logging.INFO, (self.access_log_format % atoms) + (' - %s %dms' % (get_request_id(), round((time.time() - response.time)*1000))) + ' X-Requested-With=' + inheaders.get('X-Requested-With'))

The X-Requested-With header gets added to the end of the line nicely, so I expect the same when you have the X-Forwarded-For header present.

Note, changes made down there likely won't be supported and certainly won't survive a Splunk update. Any modifications require a restart of Splunkweb.

[andrewcg]

Wouldn't it be better to change line 38 from:

    atoms = {'h': remote.name or remote.ip,

To:

    atoms = {'h': inheaders.get('X-Forwarded-For', '') or remote.name or remote.ip,

This will put the X-Forwarded-For header in the h atom if it exists. That way any Splunk App, dashboard or search that is looking at these logs would not need to be updated to look of the client IP at the end of the log line.

Also the location of the file to edit is here, at least in Splunk 6.3 for 64 bit Linux:

$SPLUNK_HOME/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/customlogmanager.py

[andrewcg]

This seems like a critical feature for Search Head clusters. If you have a load balancer or if you offload your SSL you really need the XForward header in order to know where users are coming from.

[jimcroft]

Yeah, that's not pretty but thanks for the pointer!

[ttsarenko_splun]

Still works in 2021, except look in python3.x folder instead of 2