Alerting

Why is my real-time search correlation for SSH detecting false positives with user sshd?

reljssplunk
Engager

After searching for a bit, I can't find an exact fix to this issue--
I'm having some weird edge cases with a realtime search that we're using to alert on SSH logins.

Under normal circumstances the search below triggers an alert if there is a successful login by a user after 4 failed logins in 5 minutes. Normally this works fine, however yesterday we showed brute force attempts from someone trying to log in as a user called "sshd".

This caused the search to trigger for any users successfully logging in within 5 minutes of the attempt, I'm assuming because it interpeted their successful authentication attempts as successes by the "sshd" user.

Is there any good way to modify my search to account for this?

Search code:

ssh* authenticated [search ssh* "authentication failure" | rex field=_raw "rhost=(?<ipfrom>.*) user=(?<username>.*)" | stats count by username | where count >= 4 | rename username AS query | fields query ]

Sanitized logs:

Apr 23 14:15:55 hostX auth.debug<39>: sshd[12243]: debug1: monitor_child_preauth: bob has been authenticated by privileged process
Apr 23 14:15:53 hostX authpriv.info<86>: sshd[12245]: pam_krb5(sshd:auth): user bob authenticated as bob@myhost.net
Apr 23 14:12:27 hostX authpriv.notice<85>: sshd[15524]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=100.100.100.555 user=sshd
Apr 23 14:12:13 hostX authpriv.notice<85>: sshd[15524]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=100.100.100.555 user=sshd
Apr 23 14:11:44 hostX authpriv.notice<85>: sshd[15440]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=100.100.100.555 user=sshd
Apr 23 14:10:32 hostX authpriv.notice<85>: sshd[15440]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=100.100.100.555 user=sshd
0 Karma

aweitzman
Motivator

Your problem is that the sshd returned from your subsearch appears to be matching the sshd in the name of the process, not the user name.

To fix this, you need to rex your event into fields, assign the username to a field name, and then at the end of your subsearch, assign the result to that field name rather than query. That ought to get you what you want.

Get Updates on the Splunk Community!

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...