topic Re: Anyway to see X-Forwarded-For in Splunk access logs? in Security

Anyway to see X-Forwarded-For in Splunk access logs?

jimcroft — Tue, 25 Feb 2014 17:32:11 GMT

I'm looking to implement some brute force login attempt detection/mitigation for a Splunk cluster. Splunkweb in this case is running behind a loadbalancer so I can't use the source IP from the access logs. I need to rely on the X-Forwarded-For header in the HTTP request but that's not being logged.

Does anyone know how to make splunkweb log the X-Forwarded-For in its access logs?

Thanks

Jim

Re: Anyway to see X-Forwarded-For in Splunk access logs?

martin_mueller — Tue, 25 Feb 2014 20:16:12 GMT

Yeah... just not a very nice one.

In $SPLUNK_HOME/Python-2.7/Lib/site-packages/splunk/appserver/mrsparkle/lib/customlogmanager.py's access() method you can fiddle with the log format as you please.
To verify this I've quickly appended line 63 to get this:

self.access_log.log(logging.INFO, (self.access_log_format % atoms) + (' - %s %dms' % (get_request_id(), round((time.time() - response.time)*1000))) + ' X-Requested-With=' + inheaders.get('X-Requested-With'))

The X-Requested-With header gets added to the end of the line nicely, so I expect the same when you have the X-Forwarded-For header present.

Note, changes made down there likely won't be supported and certainly won't survive a Splunk update. Any modifications require a restart of Splunkweb.

Re: Anyway to see X-Forwarded-For in Splunk access logs?

jimcroft — Wed, 26 Feb 2014 09:44:11 GMT

Yeah, that's not pretty but thanks for the pointer!

Re: Anyway to see X-Forwarded-For in Splunk access logs?

andrewcg — Tue, 25 Aug 2015 15:14:00 GMT

This seems like a critical feature for Search Head clusters. If you have a load balancer or if you offload your SSL you really need the XForward header in order to know where users are coming from.

Re: Anyway to see X-Forwarded-For in Splunk access logs?

andrewcg — Fri, 02 Oct 2015 20:16:43 GMT

Wouldn't it be better to change line 38 from:

    atoms = {'h': remote.name or remote.ip,

To:

    atoms = {'h': inheaders.get('X-Forwarded-For', '') or remote.name or remote.ip,

This will put the X-Forwarded-For header in the h atom if it exists. That way any Splunk App, dashboard or search that is looking at these logs would not need to be updated to look of the client IP at the end of the log line.

Also the location of the file to edit is here, at least in Splunk 6.3 for 64 bit Linux:

$SPLUNK_HOME/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/customlogmanager.py

Re: Anyway to see X-Forwarded-For in Splunk access logs?

maraman_splunk — Thu, 20 Jul 2017 14:40:32 GMT

Hi,

has somebody made this work with 6.5 or 6.6 ?
I don't see any effect after modifying this file so I'm not sure it's still used by these versions in default webserver configuration ...

Re: Anyway to see X-Forwarded-For in Splunk access logs?

gjanders — Thu, 20 Jul 2017 22:21:35 GMT

I tried two or three different files that mentioned access_log/atoms and I was unable to get this working...

In fact I couldn't even get the log to update to write the new log statement I had added so I'm unsure which file controls the access log in the newer 6.5.x versions.

Re: Anyway to see X-Forwarded-For in Splunk access logs?

mtulett_splunk — Wed, 17 Jan 2018 02:05:42 GMT

For any visitors finding this in 2018: The solution from andrewcg's comment above is still working for me in 7.0.

Re: Anyway to see X-Forwarded-For in Splunk access logs?

gjanders — Wed, 17 Jan 2018 02:42:23 GMT

Thanks for the confirmation, I might try again in Splunk 7...

Re: Anyway to see X-Forwarded-For in Splunk access logs?

m1k34Splunk — Wed, 07 Feb 2018 13:52:24 GMT

Just for reference, I've successfully tested the andrewcg solution in 6.6.5.
P.S. Watch out for the correct identation or it won't work.

Re: Anyway to see X-Forwarded-For in Splunk access logs?

john_thom — Mon, 12 Feb 2018 22:25:13 GMT

Hey andrewcg or m1k34Splunk,
Did you make this change on the search heads themselves? I did it on my search heads and it doesn't appear to be working. I tried including it in atoms['h'] and the access_log write itself (I even tried injecting it into both spots once). Restarted splunkweb after each change but neither seems to have any effect. Still shows the VIP from my load balancer. I'm also on 6.6.x. I feel like I must be missing something silly here.
Thanks!

Re: Anyway to see X-Forwarded-For in Splunk access logs?

andrewcg — Tue, 29 Sep 2020 18:06:40 GMT

So it looks like the problem with this hack is that it only fixes the logging in:

web_access.log

The other logs are all unchanged:

splunkd_ui_access.log
splunkd_access.log

For example:

splunkd_ui_access.log - shows the proxy

var/log/splunk/splunkd_ui_access.log:127.0.0.1 - admin [13/Feb/2018:11:00:47.110 -0500] "GET /en-US/ HTTP/1.1" 303 105 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - c0027a96a8de667c7653660d00411b6e 18ms
var/log/splunk/splunkd_ui_access.log:127.0.0.1 - admin [13/Feb/2018:11:00:47.183 -0500] "GET /en-US/app/launcher HTTP/1.1" 303 110 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - c0027a96a8de667c7653660d00411b6e 181ms
var/log/splunk/splunkd_ui_access.log:127.0.0.1 - admin [13/Feb/2018:11:00:47.516 -0500] "GET /en-US/app/launcher/home HTTP/1.1" 200 1264 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - c0027a96a8de667c7653660d00411b6e 161ms

web_access.log - shows the X-Forwarded-For

var/log/splunk/web_access.log:10.32.136.60 - admin [13/Feb/2018:11:00:47.111 -0500] "GET /en-US/ HTTP/1.1" 303 105 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - 5a830baf1c7ff401228990 17ms
var/log/splunk/web_access.log:10.32.136.60 - admin [13/Feb/2018:11:00:47.183 -0500] "GET /en-US/app/launcher HTTP/1.1" 303 110 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - 5a830baf2f7ff401228a10 180ms
var/log/splunk/web_access.log:10.32.136.60 - admin [13/Feb/2018:11:00:47.517 -0500] "GET /en-US/app/launcher/home HTTP/1.1" 200 1264 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - 5a830baf847ff4012285d0 159ms

Re: Anyway to see X-Forwarded-For in Splunk access logs?

john_thom — Tue, 29 Sep 2020 18:06:48 GMT

Ahh, thank you! This is very helpful. I was using the following search which is only showing the LB IP address:
index=_audit action="login attempt" info=failed

When I look at the web_access logs I do indeed see the correct information now:
index=_internal source="*web_access.log"

I'll see if I can poke around today and figure out if there is any way to modify the login attempt entries to check for an XFF header first and if I find anything I'll share with the community.

Thanks for the help, andrewcg!

Re: Anyway to see X-Forwarded-For in Splunk access logs?

hello007 — Thu, 25 Apr 2019 02:26:46 GMT

The log of login attempt action comes from $SPLUNK_HOME/bin/splunkd, there is no other way to modify it for showing the XFF msg.

The web_access.log only log the requests which after login, not the fail login attempt.

Re: Anyway to see X-Forwarded-For in Splunk access logs?

ttsarenko_splun — Fri, 30 Jul 2021 04:39:34 GMT

Still works in 2021, except look in python3.x folder instead of 2