Need to make advanced query for Cisco Ironport Logs

[balbano]

Hi All,

For the past few months I have been testing the DLP Feature of the Cisco Ironport to help block any sensitive data (i.e. credit card numbers) from being sent.

I have been indexing these logs in Splunk and want to make an alert to send to system administrators if a server tries to send an email with a credit card number. (Obviously an alert to the sender will not work in this case as the sender is a server. )

However, there does not appear to be a log line that will indicate a DLP Policy Violation between a sender and receiver. They are separated in multiple log lines like so:

2011 Oct 28 16:13:51 ironport_server mail [info] splunk_maillogs:nopid Info: MID 6509657 quarantined to "Policy" (DLP violation)

2011 Oct 28 16:13:51 ironport_server mail [info] splunk_maillogs:nopid Info: MID 6509657 DLP violation

2011 Oct 28 16:13:38 ironport_server mail [info] splunk_maillogs:nopid Info: MID 6509657 ready 53 bytes from brian@server

2011 Oct 28 16:13:38 ironport_server mail [info] splunk_maillogs:nopid Info: MID 6509657 ICID 6237668 RID 0 To: brian@testsite

I guess the only key here to bind these two types of log messages together is with the message ID.

However, this would need to be a 2 part query:

1. Find any log with a DLP Policy Violation.
2. Find all logs pertaining to the MID value specified in the Violation Log.

Any idea on how to make a 2 part query and make an alert out of it?

Or is there a better way to do this?

Let me know what you think.

Any help is appreciated.

Thanks.

Brian

[mcronkrite_splu]

The transaction on MID isn't enough, because the different models of ESA spawn new MID's and reference the original MID as "internal_message_id". Instead, a more complex union is required to accomplish the Ironport logs self-join.

| union
  [ search index=ironport mid=* (message_size=* OR internal_message_id=* OR sender=* OR recipient=* OR subject=*)
    | eval message_size_mb=(message_size_mb/1024/1024)
    | fields mid, message_size_mb, internal_message_id, sender, recipient, subject, _time
    | stats min(_time) as _time values(*) as * by mid]
  [ search index=ironport file_name=*
    | rename mid as internal_message_id
    | stats min(_time) as _time values(file_name) as file_names by internal_message_id
    | fields internal_message_id file_names _time]
| stats values(*) as * min(_time) as _time by internal_message_id
| search mid=* message_size_mb=*
| table _time, mid, internal_message_id, sender, subject, recipient, message_size_mb, file_names
| collect index=ironport sourcetype=ironport:summary addtime=false

This search will look at ironport logs in the index=ironport which have MID=* and other key values. Then using this as a union change the MID as internal_message_ID and search again for associated records. Finally, collect the data into a new sourcetype=ironport:summary and use the original data time for the summary to preserve the event's original time. Run this search as a scheduled search to populate email data either for a data model or to query directly on the new summarized information.

[imarks004 · Accepted solution]

I finally got around to posting to Answers how I do this without the DLP license. Basic Cisco Ironport ESA DLP Monitoring Without DLP License

This ends up generating a single event into the cisco_esa sourcetype containing the original MID, signature (ie. credit_card_violation), sender, recipient, subject, attachment details, and a few other fields.

Having a single event makes the search much faster than using a transaction, but since it does show the original MID, it will show up in a single transaction search if you need additional data without needing a subsearch or join.

You might be able tweak the way the DLP event logs using the format from my example to get all the details in one event.

[balbano]

Thanks imarks!!! This works for me. I already have the RSA DLP License but I wanted to be able to setup logging with all the info I needed in one line so I can create custom alerts. This gets it done for me. Thanks again!!! - Brian

[kristian_kolb]

Have you looked at the transaction command? Assuming that you extract the message ID as MID;

sourcetype=<your_source> | transaction MID | search "DLP Violation"

This will basically group all messages based on the MID into one event, and then search these grouped events for the string "DLP Violation". These can then be reported/alerted on.

The downside is that the transaction command is rather expensive, and mail logs tend to be noisy. If the log file/sourcetype ONLY contains violations, and not all SMTP traffic, it may still be ok though. There are a few parameters to the transaction command you might want to check out, such as how far apart (in time) two messages can be while still being considered part of the same transaction.

For more information, look at:

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Transaction

Hope this helps,

Kristian