Getting Cisco Ironport ESA data into the Common Information Model

[kalifehj]

I am having a tough time understanding how anyone is getting Cisco Ironport ESA data to map to the CIM for use in things like Enterprise Security. Where I work, I would say that email is the most likely vector of malware and/or phishing schemes that attempt to get credentials. Of course I want my ESA data to get matched to the threat intelligence feeds and notable events, but I haven't been able to get an answer from either official Splunk support, or even from vendors that want to sell me their threat intelligence platforms and/or other security tools.

In order to map email to the CIM, you at least have to have the points of data to match on. Src, dest, src_user, recipient, subject, file_name, url, filtering signatures and the all important action/filter_action. This data gives you the IOC matching points. The problem is that the Cisco ESA logs are sent to Splunk in a way that does not allow for easy recognition of all those points in a single "event".

Here is an example of a Cisco ESA "mail event", this was pulled from the Cisco ESA console's "message tracking" feature. This is the only place that I know how to get this complete of a picture. Otherwise by looking at the raw logs, each email flow process has events that are completely out of order, there is no expectation of being able to put them together with a single "mail ID".

13 Aug 2016 07:28:48 (GMT -04:00) Protocol SMTP interface CS DMZ (IP X.X.X.X) on incoming connection (ICID 122155365) from sender IP X.X.X.X. Reverse DNS host mail-relay-02.XXXXXX.com verified yes.
13 Aug 2016 07:28:48 (GMT -04:00) (ICID 122155365) ACCEPT sender group ACCEPTLIST match sbrs[1.0:10.0] SBRS 3.5
13 Aug 2016 07:28:48 (GMT -04:00) Incoming connection (ICID 122155365) successfully accepted TLS protocol (UNKNOWN:302) cipher DHE-RSA-AES256-SHA.
13 Aug 2016 07:28:49 (GMT -04:00) Start message 56789163 on incoming connection (ICID 122155365).
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 enqueued on incoming connection (ICID 122155365) from notifications@XXXXX.com.
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 on incoming connection (ICID 122155365) added recipient (Jeff.Kalifeh@XXXXXXX.com).
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 contains message ID header '<57af0470aa492_b1a211d933021393@XXXX.mail>'.
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 original subject on injection: Daily Recap for Friday, August 12
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 (19159 bytes) from notifications@XXXX.com ready.
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 matched per-recipient policy Test for inbound mail policies.
13 Aug 2016 07:28:49 (GMT -04:00) SMTP delivery connection (DCID 35832763) opened from Cisco IronPort interface X.X.X.X to IP address X.X.X.X on port 25.
13 Aug 2016 07:28:49 (GMT -04:00) Delivery connection (DCID 35832763) successfully accepted TLS protocol TLSv1 cipher ECDHE-RSA-AES256-SHA .
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 scanned by Anti-Spam engine: CASE. Interim verdict: Negative
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 scanned by Anti-Spam engine CASE. Interim verdict: definitely negative.
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 scanned by Anti-Spam engine: CASE. Final verdict: Negative
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 scanned by Anti-Virus engine Sophos. Interim verdict: CLEAN
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 scanned by Anti-Virus engine. Final verdict: Negative
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 scanned by Advanced Malware Protection engine. Final verdict: CLEAN
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 scanned by Outbreak Filters. Verdict: Negative
13 Aug 2016 07:28:49 (GMT -04:00) Message 56789163 queued for delivery.
13 Aug 2016 07:28:49 (GMT -04:00) (DCID 35832763) Delivery started for message 56789163 to Jeff.Kalifeh@XXXXXXX.com.
13 Aug 2016 07:28:50 (GMT -04:00) (DCID 35832763) Delivery details: Message 56789163 sent to Jeff.Kalifeh@XXXXXX.com
13 Aug 2016 07:28:50 (GMT -04:00) Message 56789163 to Jeff.Kalifeh@mXXXXX.com received remote SMTP response '2.6.0 <57af0470aa492_b1a211d933021393@jobs-13.mail> [InternalId=72988006] Queued mail for delivery'.

( I X'ed out the domains and IPs.) So the typical method is to look for the MIDs, ICIDs, DCIDs, and attempt to weave them together to recreate this view. This could include some transactions, eval stuff, eventstats stuff, etc. But even then, the CIsco ESA mail flow process constantly rewrites the MIDs as certain things happen, such as matching a DLP rule, matching some content filtering rule, etc. Also keep in mind that the DCIDs and ICIDs can be different values for each MID depending on how many recipients the message went to or was sent to.

So determining if that "known bad" email spammer was successful in delivering a malware campaign to your organization becomes impossible. You need to know all the values of the "event", you need to see if it was blocked/allowed, you need to see the src/dest/everything to make that call. I am hoping that someone has figured this out, I mean Splunk ES basically requires having this figured out in order to provide visibility into Cisco ESA email events.

[ferozkhanpeermo]

You just have to add the SPlunk_TA_Cisco-esa add on your HWF [if you are collecting the data in your HWF] & to all your indexer nodes in your cluster and to your Search Head Cluster. If you are having a dedicated SearchHead for your Splunk ES module, you need to add the add-on there also. Only then the events coming from Cisco-ESA appliance will be transformed when you do the search in Splunk ES module

[mikaelbje]

A complete add-on that does all of this is posted here:

https://github.com/inspired/TA-cisco-esa-extras

It requires the Splunk Add-on for Cisco ESA, but adds the extra logic needed to group the events into a summary.

[gfuente · Accepted solution]

Hello

You could schedule the transaction search and collect the data into a summary index, and then add the data in the summary index to be included in the data model.

Regards

[kalifehj]

This is the answer. Group it however, but make a summary index from the results.

[mikaelbje]

Hi kalifehj,

any chance you could share the summary index search you created? I'm having the same issue, but would also like to make AMP events compliant with the CIM Malware data model.

[gfuente]

Thanks

Please mark it as solved

Regards

[kalifehj]

I am not sure if anyone is as interested in this as I am but, I was able to find a way to make this work. The solution that I found involves adding a custom log entry action to the email message filter feature within Ironport. This allows you to then use a combo of eventstats + stats to group the messages as a data model constraints for CIM email.

Either no one who has this working decided to respond to this post, or no one using Cisco Ironport ESA has the CIM email data model mapped.

[talonso]

Hola, que sourcetype utilizó??

[Richfez]

A partial, potential answer to start from.

I added your log bits as a new input to my test system. I let Splunk autodetect whatever it found (timestamps only) and saved that as sourcetype answers_441191. I think everything else was left at defaults.

From that, the following search creates a transaction out of all 20 events dynamically.

source="mysource.txt" host="mytestbox" index="temp_441191" sourcetype="answers_441191" 
| rex field=_raw "DCID\s(?<dcid>[^\)]+)" 
| rex field=_raw "ICID\s(?<icid>[^\)]+)" 
| rex field=_raw "[Mm]essage\s(?<mcid>\S+)"
| rex field=_raw "\<(?<iid>[^@]+)@"
| eval connector=mvappend(dcid, idic, mcid, iid)
| transaction connector

If you convert all those to props/transforms as appropriate, add in all the individual CIM fields (like a rex recipient\s\((?<recipient>[^\))) (just a guess) for recipient and whatnot, then .. well, I am not aware of how to do the mvappend or transaction "behind the scenes" in the way you'd want, but at least it's a start.

But still, hopefully this will provide enough to get you moving in the right direction. This example should mostly run on moderate sized data sets as is as long as you modify the base search to get your data.

[kalifehj]

Thanks for the response! So something like this is what I have come up with for grouping the one line events into a group that represents an "email event". The issue is that how do I turn that into the data that feeds into the email data model? I am guessing I can run a scheduled search and tag the results with the correct tag for CIM email. Is that the best way to go from an ad-hoc search into the data model

[kbrown_splunk]

You are using the Cisco ESA CIM compliant app correct?
https://splunkbase.splunk.com/app/1761/
Are you having issues even though you are using the app?

[kalifehj]

Yes, I am using the Cisco ESA CIM TA. The TA is very helpful for parsing out the fields, but the events still come in as multiple, separate single lines. one event might have the src_ip, and then 10 rows down you might get the event that contains the sender and subject. Rich7177's response above is a method of weaving together the single line events into a single "email event". But I am still trying to figure out how to take that search and turn it into a data model.

[Richfez]

Oh, I should've checked. That's a much better answer if it'll work. 🙂