Splunk Search

Correlate extracted banned IP field with corresponding SSH log events

burns498
Explorer

Hello Splunk Community,

I am new to Splunk so please bear with me. My end goal is to construct a dashboard summary of our fail2ban intrusion prevention framework. The current panel I am attempting to construct should correlate the list of temporarily banned IPs from fail2ban with the corresponding SSH logs. I could then for example pull out the attempted usernames or the total count of SSH attempts from a particular IP.

A sample of a fail2ban log and a corresponding SSH logs:

May 28 11:39:33 [hostname] fail2ban.actions: WARNING [ssh] Ban 42.121.111.136
May 28 11:39:32 [hostname] sshd[9748]: Failed password for invalid user HonestQiao from 42.121.111.136 port 14130 ssh2
May 28 11:39:28 [hostname] sshd[9746]: Failed password for invalid user 5s1admin from 42.121.111.136 port 13918 ssh2
May 28 11:39:24 [hostname] sshd[9744]: Failed password for root from 42.121.111.136 port 13734 ssh2

I can retrieve all of the desired events and extract the banned IPs with the following search:

(process=sshd "failed password") OR process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?<fail2ban_ip>\d+\.\d+\.\d+\.\d+)"

I then want to drill down to only viewing the SSH logs from the banned IPs. I believe this can be done with the transaction command but my attempts to filter by fail2ban_ip have proven fruitless. I tried:

... | transaction fail2ban_ip

which groups by banned IP fine but only shows the fail2ban logs instead of SSH. Using "src_ip" or "fail2ban_ip, src_ip" as the transaction field list shows SSH logs for all IPs (including unbanned) or an empty result respectively.

I browsed quite a few transaction examples but couldn't find something that helped me out. Any advice on how to use transaction or how to better approach the problem would be greatly appreciated.

0 Karma
1 Solution

Ayn
Legend

Sounds like using a subsearch could be appropriate in your scenario? You'd run the subsearch over the fail2ban data in order to extract the IP addresses, then use that list to filter out events from those IP's in your SSH logs. Something like this:

(process=sshd "failed password") [search process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?<fail2ban_ip>\d+\.\d+\.\d+\.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip]

What will happen is the subsearch (within the square brackets) will run first of all and compile a list of IP's that is then expanded into a filter string for the outer search command. So,

(process=sshd "failed password") ((src_ip="ip1") OR (src_ip="ip2") OR ... ))

You can see a similar example of how to use subsearches here: http://docs.splunk.com/Documentation/Splunk/5.0.2/Tutorial/Useasubsearch

View solution in original post

Ayn
Legend

Sounds like using a subsearch could be appropriate in your scenario? You'd run the subsearch over the fail2ban data in order to extract the IP addresses, then use that list to filter out events from those IP's in your SSH logs. Something like this:

(process=sshd "failed password") [search process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?<fail2ban_ip>\d+\.\d+\.\d+\.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip]

What will happen is the subsearch (within the square brackets) will run first of all and compile a list of IP's that is then expanded into a filter string for the outer search command. So,

(process=sshd "failed password") ((src_ip="ip1") OR (src_ip="ip2") OR ... ))

You can see a similar example of how to use subsearches here: http://docs.splunk.com/Documentation/Splunk/5.0.2/Tutorial/Useasubsearch

burns498
Explorer

Perfect! Piping the fields command at the end works great. Thank you!

0 Karma

Ayn
Legend

Ohh - sorry! Forgot one very important thing at the end of the subsearch - fields src_ip which makes the subsearch emit a filter string ONLY for src_ip and nothing else. So:

(process=sshd "failed password") [search process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?<fail2ban_ip>\d+\.\d+\.\d+\.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip | fields src_ip]

burns498
Explorer

Running "process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?\d+.\d+.\d+.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip" results in the correct distinct events and the src_ip field is generated correctly.

Running "(process=sshd "failed password") [search ... ]" results in nothing.

Running "process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?\d+.\d+.\d+.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip | format" results in a long string.
"( ( date_hour="13" AND date_mday="29" AND date_ ... "

0 Karma

Ayn
Legend

If you run the subsearch on its own and append "| format" at the end, then you'll get the exact filter string that the subsearch outputs. Does that filter string look sane?

0 Karma

burns498
Explorer

Ayn,
Thank you for the response. This answer makes sense to me but I am not receiving output as expected. The entire command results in no matching events. Just the subsearch works fine. It appears that the syntax is correct though. Any ideas?

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...