Splunk Search

Correlate extracted banned IP field with corresponding SSH log events

Explorer

Hello Splunk Community,

I am new to Splunk so please bear with me. My end goal is to construct a dashboard summary of our fail2ban intrusion prevention framework. The current panel I am attempting to construct should correlate the list of temporarily banned IPs from fail2ban with the corresponding SSH logs. I could then for example pull out the attempted usernames or the total count of SSH attempts from a particular IP.

A sample of a fail2ban log and a corresponding SSH logs:

May 28 11:39:33 [hostname] fail2ban.actions: WARNING [ssh] Ban 42.121.111.136
May 28 11:39:32 [hostname] sshd[9748]: Failed password for invalid user HonestQiao from 42.121.111.136 port 14130 ssh2
May 28 11:39:28 [hostname] sshd[9746]: Failed password for invalid user 5s1admin from 42.121.111.136 port 13918 ssh2
May 28 11:39:24 [hostname] sshd[9744]: Failed password for root from 42.121.111.136 port 13734 ssh2

I can retrieve all of the desired events and extract the banned IPs with the following search:

(process=sshd "failed password") OR process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?<fail2ban_ip>\d+\.\d+\.\d+\.\d+)"

I then want to drill down to only viewing the SSH logs from the banned IPs. I believe this can be done with the transaction command but my attempts to filter by fail2ban_ip have proven fruitless. I tried:

... | transaction fail2ban_ip

which groups by banned IP fine but only shows the fail2ban logs instead of SSH. Using "src_ip" or "fail2ban_ip, src_ip" as the transaction field list shows SSH logs for all IPs (including unbanned) or an empty result respectively.

I browsed quite a few transaction examples but couldn't find something that helped me out. Any advice on how to use transaction or how to better approach the problem would be greatly appreciated.

0 Karma
1 Solution

Legend

Sounds like using a subsearch could be appropriate in your scenario? You'd run the subsearch over the fail2ban data in order to extract the IP addresses, then use that list to filter out events from those IP's in your SSH logs. Something like this:

(process=sshd "failed password") [search process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?<fail2ban_ip>\d+\.\d+\.\d+\.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip]

What will happen is the subsearch (within the square brackets) will run first of all and compile a list of IP's that is then expanded into a filter string for the outer search command. So,

(process=sshd "failed password") ((src_ip="ip1") OR (src_ip="ip2") OR ... ))

You can see a similar example of how to use subsearches here: http://docs.splunk.com/Documentation/Splunk/5.0.2/Tutorial/Useasubsearch

View solution in original post

Legend

Sounds like using a subsearch could be appropriate in your scenario? You'd run the subsearch over the fail2ban data in order to extract the IP addresses, then use that list to filter out events from those IP's in your SSH logs. Something like this:

(process=sshd "failed password") [search process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?<fail2ban_ip>\d+\.\d+\.\d+\.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip]

What will happen is the subsearch (within the square brackets) will run first of all and compile a list of IP's that is then expanded into a filter string for the outer search command. So,

(process=sshd "failed password") ((src_ip="ip1") OR (src_ip="ip2") OR ... ))

You can see a similar example of how to use subsearches here: http://docs.splunk.com/Documentation/Splunk/5.0.2/Tutorial/Useasubsearch

View solution in original post

Explorer

Perfect! Piping the fields command at the end works great. Thank you!

0 Karma

Legend

Ohh - sorry! Forgot one very important thing at the end of the subsearch - fields src_ip which makes the subsearch emit a filter string ONLY for src_ip and nothing else. So:

(process=sshd "failed password") [search process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?<fail2ban_ip>\d+\.\d+\.\d+\.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip | fields src_ip]

Explorer

Running "process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?\d+.\d+.\d+.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip" results in the correct distinct events and the src_ip field is generated correctly.

Running "(process=sshd "failed password") [search ... ]" results in nothing.

Running "process=fail2ban.actions | rex "fail2ban.actions: WARNING .* Ban (?\d+.\d+.\d+.\d+)" | dedup fail2ban_ip | rename fail2ban_ip as src_ip | format" results in a long string.
"( ( date_hour="13" AND date_mday="29" AND date_ ... "

0 Karma

Legend

If you run the subsearch on its own and append "| format" at the end, then you'll get the exact filter string that the subsearch outputs. Does that filter string look sane?

0 Karma

Explorer

Ayn,
Thank you for the response. This answer makes sense to me but I am not receiving output as expected. The entire command results in no matching events. Just the subsearch works fine. It appears that the syntax is correct though. Any ideas?

0 Karma
State of Splunk Careers

Access the Splunk Careers Report to see real data that shows how Splunk mastery increases your value and job satisfaction.

Find out what your skills are worth!