Security

Anyway to see X-Forwarded-For in Splunk access logs?

jimcroft
Explorer

I'm looking to implement some brute force login attempt detection/mitigation for a Splunk cluster. Splunkweb in this case is running behind a loadbalancer so I can't use the source IP from the access logs. I need to rely on the X-Forwarded-For header in the HTTP request but that's not being logged.

Does anyone know how to make splunkweb log the X-Forwarded-For in its access logs?

Thanks

Jim

1 Solution

martin_mueller
SplunkTrust
SplunkTrust

Yeah... just not a very nice one.

In $SPLUNK_HOME/Python-2.7/Lib/site-packages/splunk/appserver/mrsparkle/lib/customlogmanager.py's access() method you can fiddle with the log format as you please.
To verify this I've quickly appended line 63 to get this:

self.access_log.log(logging.INFO, (self.access_log_format % atoms) + (' - %s %dms' % (get_request_id(), round((time.time() - response.time)*1000))) + ' X-Requested-With=' + inheaders.get('X-Requested-With'))

The X-Requested-With header gets added to the end of the line nicely, so I expect the same when you have the X-Forwarded-For header present.

Note, changes made down there likely won't be supported and certainly won't survive a Splunk update. Any modifications require a restart of Splunkweb.

View solution in original post

maraman_splunk
Splunk Employee
Splunk Employee

Hi,

has somebody made this work with 6.5 or 6.6 ?
I don't see any effect after modifying this file so I'm not sure it's still used by these versions in default webserver configuration ...

0 Karma

m1k34Splunk
Engager

Just for reference, I've successfully tested the andrewcg solution in 6.6.5.
P.S. Watch out for the correct identation or it won't work.

0 Karma

john_thom
New Member

Hey andrewcg or m1k34Splunk,
Did you make this change on the search heads themselves? I did it on my search heads and it doesn't appear to be working. I tried including it in atoms['h'] and the access_log write itself (I even tried injecting it into both spots once). Restarted splunkweb after each change but neither seems to have any effect. Still shows the VIP from my load balancer. I'm also on 6.6.x. I feel like I must be missing something silly here.
Thanks!

0 Karma

andrewcg
Path Finder

So it looks like the problem with this hack is that it only fixes the logging in:

  • web_access.log

The other logs are all unchanged:

  • splunkd_ui_access.log
  • splunkd_access.log

For example:

splunkd_ui_access.log - shows the proxy

var/log/splunk/splunkd_ui_access.log:127.0.0.1 - admin [13/Feb/2018:11:00:47.110 -0500] "GET /en-US/ HTTP/1.1" 303 105 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - c0027a96a8de667c7653660d00411b6e 18ms
var/log/splunk/splunkd_ui_access.log:127.0.0.1 - admin [13/Feb/2018:11:00:47.183 -0500] "GET /en-US/app/launcher HTTP/1.1" 303 110 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - c0027a96a8de667c7653660d00411b6e 181ms
var/log/splunk/splunkd_ui_access.log:127.0.0.1 - admin [13/Feb/2018:11:00:47.516 -0500] "GET /en-US/app/launcher/home HTTP/1.1" 200 1264 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - c0027a96a8de667c7653660d00411b6e 161ms

web_access.log - shows the X-Forwarded-For

var/log/splunk/web_access.log:10.32.136.60 - admin [13/Feb/2018:11:00:47.111 -0500] "GET /en-US/ HTTP/1.1" 303 105 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - 5a830baf1c7ff401228990 17ms
var/log/splunk/web_access.log:10.32.136.60 - admin [13/Feb/2018:11:00:47.183 -0500] "GET /en-US/app/launcher HTTP/1.1" 303 110 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - 5a830baf2f7ff401228a10 180ms
var/log/splunk/web_access.log:10.32.136.60 - admin [13/Feb/2018:11:00:47.517 -0500] "GET /en-US/app/launcher/home HTTP/1.1" 200 1264 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0" - 5a830baf847ff4012285d0 159ms
0 Karma

john_thom
New Member

Ahh, thank you! This is very helpful. I was using the following search which is only showing the LB IP address:
index=_audit action="login attempt" info=failed

When I look at the web_access logs I do indeed see the correct information now:
index=_internal source="*web_access.log"

I'll see if I can poke around today and figure out if there is any way to modify the login attempt entries to check for an XFF header first and if I find anything I'll share with the community.

Thanks for the help, andrewcg!

0 Karma

hello007
New Member

The log of login attempt action comes from $SPLUNK_HOME/bin/splunkd, there is no other way to modify it for showing the XFF msg.

The web_access.log only log the requests which after login, not the fail login attempt.

0 Karma

gjanders
SplunkTrust
SplunkTrust

I tried two or three different files that mentioned access_log/atoms and I was unable to get this working...

In fact I couldn't even get the log to update to write the new log statement I had added so I'm unsure which file controls the access log in the newer 6.5.x versions.

0 Karma

mtulett_splunk
Splunk Employee
Splunk Employee

For any visitors finding this in 2018: The solution from andrewcg's comment above is still working for me in 7.0.

gjanders
SplunkTrust
SplunkTrust

Thanks for the confirmation, I might try again in Splunk 7...

0 Karma

martin_mueller
SplunkTrust
SplunkTrust

Yeah... just not a very nice one.

In $SPLUNK_HOME/Python-2.7/Lib/site-packages/splunk/appserver/mrsparkle/lib/customlogmanager.py's access() method you can fiddle with the log format as you please.
To verify this I've quickly appended line 63 to get this:

self.access_log.log(logging.INFO, (self.access_log_format % atoms) + (' - %s %dms' % (get_request_id(), round((time.time() - response.time)*1000))) + ' X-Requested-With=' + inheaders.get('X-Requested-With'))

The X-Requested-With header gets added to the end of the line nicely, so I expect the same when you have the X-Forwarded-For header present.

Note, changes made down there likely won't be supported and certainly won't survive a Splunk update. Any modifications require a restart of Splunkweb.

andrewcg
Path Finder

Wouldn't it be better to change line 38 from:

    atoms = {'h': remote.name or remote.ip,

To:

    atoms = {'h': inheaders.get('X-Forwarded-For', '') or remote.name or remote.ip,

This will put the X-Forwarded-For header in the h atom if it exists. That way any Splunk App, dashboard or search that is looking at these logs would not need to be updated to look of the client IP at the end of the log line.

Also the location of the file to edit is here, at least in Splunk 6.3 for 64 bit Linux:

$SPLUNK_HOME/lib/python2.7/site-packages/splunk/appserver/mrsparkle/lib/customlogmanager.py

andrewcg
Path Finder

This seems like a critical feature for Search Head clusters. If you have a load balancer or if you offload your SSL you really need the XForward header in order to know where users are coming from.

jimcroft
Explorer

Yeah, that's not pretty but thanks for the pointer!

0 Karma

ttsarenko_splun
Splunk Employee
Splunk Employee

Still works in 2021, except look in python3.x folder instead of 2

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...