Splunk Enterprise Security - Host Sending Excessive Emails Alert / Email Data model

[kamoenix]

Hi Everyone

I'm having trouble with one of the alerts in Enterprise Security which is causing a lot of noise and false positives. I've tuned the correlation rule to where I want it, but the problem is that the count where it shows how many emails were sent to a recipient is broken. Not sure if this search comes with ES or not but in the data model, it shows that this is a calculated field for recipient_count.

case(isnum(recipient_count),recipient_count,,isnotnull(recipient),mvcount(recipient),1=1,1)

The problem with the search above is that it's counting events which aren't relevant and therefore are skewing our results.

I've tried numerous different ways of excluding the results in the calculated fields, but it just breaks the search. I was wondering if anyone had any experience of using calculated fields above so it excluded the following items.

source_id=STOREDRIVER
directionality=originating 

Any help would be greatly appreciated.

[woodcock · Accepted solution]

You should not change the datamodel definition (for many very good reasons). The thing that may be thwarting you is the order of search time operations. See here:

https://docs.splunk.com/Documentation/Splunk/latest/Knowledge/Searchtimeoperationssequence

The best way to get enhanced context into a CIM datamodel without changing the JSON definition is to create a new tag value that you can user to filter in your search. So you should create a tag called originating with a definition of source_id=STOREDRIVER and another called STOREDRIVER with a definition of directionality=originating (the names are arbitrary) and export these definitions to global scope (so that the CIM ADM searches can see them). You also need to modify the whitelist for that datamodel to add these 2 new tags if you are on v7.*.
Then in your search go to the WHERE section of your alert's saved search and add AND NOT (YourDataModelNameHere.tag=="originating" AND YourDataModelNameHere.tag=="STOREDRIVER")

[kamoenix]

Hi @woodcock ,

Thanks for your reply. I completely agree that the order of search time operations is what is causing us issues, although I'm still unsure if what you've suggested will combat the problem.

Please correct me if I'm wrong, but if the calculated field is created before the tags, won't the search be the same?

My correlation search is below, which may add some more context to the problem.

| tstats allow_old_summaries=true sum(All_Email.recipient_count) as count,dc(All_Email.dest) as dest_count from datamodel=Email.All_Email where (All_Email.src_category!="email_servers" OR All_Email.src_category!=* AND All_Email.directionality="originating" AND NOT All_Email.source_id="STOREDRIVER") by "All_Email.src" ,_time span=1h |drop_dm_object_name("All_Email")| xswhere count from recipients_by_src_1h in email is above high OR dest_count from destinations_by_src_1h in email is above high

So from my understanding, even if I added the tags in this search, the rule would still alert because the calculated field "All_Email.recipient_count" will count the fields with the items we want to omit.

If I added the tag NOT tag=STOREDRIVER to the constraint field in the data-model (even though I know you suggested against it earlier) this will then remove the STOREDRIVER completely which will then give us a more accurate representation of the emails.

To further expand on this, when an alert appears in ES stating that there were 193 SMTP connections were observed, when taking out STOREDRIVER there are only 39 actual emails were sent, the STOREDRIVER events are actually a Microsoft Exchange process. So if we take them out of the data model completely, then recipient_count should give us a more accurate count of SMTP connections made.

[woodcock]

No, you don't change the Datamodel definition at all because the tag field is already defined in every CIM datamodel. If you just create the tags like I instructed and export them to global scope so that the CIM ADM searches will see them, then you can just use a search like this:

| tstats allow_old_summaries=true sum(All_Email.recipient_count) AS count, dc(All_Email.dest) AS dest_count
FROM datamodel=Email.All_Email
WHERE (All_Email.src_category!="email_servers" OR All_Email.src_category!=*) AND All_Email.tag="originating" AND NOT All_Email.tag="STOREDRIVER")
BY "All_Email.src" ,_time span=1h 
| drop_dm_object_name("All_Email")`` 
| xswhere count from recipients_by_src_1h in email is above high OR dest_count from destinations_by_src_1h in email is above high