Splunk Enterprise Security

Appendpipe or Multisearch for continuing processing after Outputlookup?

stroud_bc
Path Finder

I am attempting to create a custom Risk Attribution rule based on Web Proxy traffic to newly-seen (not-seen-before-yesterday) domains based off the blog post found below:

https://www.splunk.com/blog/2018/01/17/finding-new-evil-detecting-new-domains-with-splunk.html

What I am running into is the following: I want to use the "new domain monitoring" technique described in the above link, but I want to attribute risk to a USER. The problem is that I want to run my statistics WITHOUT the user field in order to update previously_seen_domains.csv and WITH the user field in order to create my risk attribution.

| tstats count from datamodel=Web where nodename=Web.Proxy by Web.url, Web.user, _time 
| rename Web.* AS * 
| eval list="mozilla" 
| `ut_parse_extended(url, list)` 
| appendpipe 
    [| stats earliest(_time) as earliest latest(_time) as latest by ut_domain
    | inputlookup append=t previously_seen_domains.csv 
    | stats min(earliest) as earliest max(latest) as latest by ut_domain
    | outputlookup previously_seen_domains.csv
    |search 1!=1
     ]
| stats earliest(_time) as earliest latest(_time) as latest by ut_domain, user 
| inputlookup append=t previously_seen_domains.csv 
| stats min(earliest) as earliest max(latest) as latest by ut_domain, user 
| eval isOutlier=if(earliest >= relative_time(now(), "-1d@d"), 1, 0) 
| convert ctime(earliest) ctime(latest) 
| where isOutlier=1

This search DOES work, but I don't know much about the internals of the |appendpipe command, and it seems like |multisearch might be more performant? Right now previously_seen_domains.csv has about 10k entries in it. If there are any suggestions on how to run the stats calculations once instead of twice with different aggregation, those would be helpful as well.

0 Karma
1 Solution

stroud_bc
Path Finder

I think I have a better understanding of |multisearch after reading through some answers on the topic. Additionally, for any future readers who are trying a similar approach, I found that the above search fails to respect the earliest values from the lookup, since the second | stats earliest(_time) as earliest latest(_time) as latest by ut_domain, user line ends up recalculating earliest values FOR EACH USER, which means the values will always fall within the relevant window and therefore will always end up attributing risk... To solve this, I turned to a much simpler and more linear approach using a subsearch with a separate |tstats command AFTER the outputlookup to join the user data to the newly-seen domain data.

View solution in original post

0 Karma

stroud_bc
Path Finder

I think I have a better understanding of |multisearch after reading through some answers on the topic. Additionally, for any future readers who are trying a similar approach, I found that the above search fails to respect the earliest values from the lookup, since the second | stats earliest(_time) as earliest latest(_time) as latest by ut_domain, user line ends up recalculating earliest values FOR EACH USER, which means the values will always fall within the relevant window and therefore will always end up attributing risk... To solve this, I turned to a much simpler and more linear approach using a subsearch with a separate |tstats command AFTER the outputlookup to join the user data to the newly-seen domain data.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Take Action Automatically on Splunk Alerts with Red Hat Ansible Automation Platform

 Are you ready to revolutionize your IT operations? As digital transformation accelerates, the demand for ...

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...