How do you search IronPort events and return the whole email transaction?

[mendesjo]

Can anyone suggest how you query IronPort logs? When I query mail logs on the ironport itself, say for an email from xzy@whatever.com, and I click on details, it groups all the events of that transaction together nicely so I can see what happened start to finish. When I search in Splunk, it doesn't do that.. so question is, how do I do that? I found one or two other folks that I think have the same problem or asking the same question as I, and the transaction command looks promising, tried a few searches, but no results. I'm fairly new to Splunk so any suggestions would be appreciated.

[mcronkrite_splu]

The transaction on MID isn't enough, because the different models of ESA spawn new MID's and reference the original MID as "internal_message_id". Instead, a more complex union is required to accomplish the Ironport logs self-join.

 | union
   [ search index=ironport mid=* (message_size=* OR internal_message_id=* OR sender=* OR recipient=* OR subject=*)
     | eval message_size_mb=(message_size_mb/1024/1024)
     | fields mid, message_size_mb, internal_message_id, sender, recipient, subject, _time
     | stats min(_time) as _time values(*) as * by mid]
   [ search index=ironport file_name=*
     | rename mid as internal_message_id
     | stats min(_time) as _time values(file_name) as file_names by internal_message_id
     | fields internal_message_id file_names _time]
 | stats values(*) as * min(_time) as _time by internal_message_id
 | search mid=* message_size_mb=*
 | table _time, mid, internal_message_id, sender, subject, recipient, message_size_mb, file_names
 | collect index=ironport sourcetype=ironport:summary addtime=false

This search will look at ironport logs in the index=ironport which have MID=* and other key values. Then using this as a union change the MID as internal_message_ID and search again for associated records. Finally, collect the data into a new sourcetype=ironport:summary and use the original data time for the summary to preserve the event's original time. Run this search as a scheduled search to populate email data either for a data model or to query directly on the new summarized information.

[MuS]

Hi mendesjo,

If you want to use the ESA add-on https://splunkbase.splunk.com/app/1761/ you must assign the sourcetype cisco_esa or cisco:esa or cisco:esa:legacy this will get you the fields which are defined in props.conf of the Add-on.
Also you can run a fast and efficient stats instead of the clumsy transaction (which will break Mapreduce) :

you're base search here | stats list(_raw) AS _raw by mid

You can learn more about the different use cases of stats here https://answers.splunk.com/answers/129424/how-to-compare-fields-over-multiple-sourcetypes-without-jo... or here http://sideviewapps.com/slides/2016_03_31_nick_mealy_grouping_talk.pptx

Hope this helps ...

cheers, MuS

[AndySplunks]

You need to use the transaction command. The explanation page for transaction includes the Cisco Ironport for example 4.

http://docs.splunk.com/Documentation/Splunk/6.3.2/SearchReference/Transaction

[trevorQmulos]

Is there any way to further filter before doing the transaction command? IE i have a subject of a spam email so I want to see the full event list. When I insert a search modifier, it takes away the transaction portion of the events.

sourcetype="cisco:esa:legacy"  | transaction mid dcid icid maxevents=10 maxspan=5s mvlist=t


I would want to add

sourcetype="cisco:esa:legacy" subject="Dogs are cool"  | transaction mid dcid icid maxevents=10 maxspan=5s mvlist=t

[javiergn]

Hi, I don't have any test ironport logs to play with but if I remember correctly you need to create transactions by MID in order to join events related to the same message ID.

Something like:

yourquery | transaction mid

If you give us more details about how your logs look like we might be able to help a bit more.

Thanks,
J

[mendesjo]

Thank you.. I tried transaction command but wasn't working. I then realized why MID wasn't working, it's because MID hasn't been defined as a field. Ok.. I'll have to figure that out. Thanks for the suggestions I'm definetly closer now.

[mendesjo]

Thanks, now the problem is, I don't have mid as field. I did install the add-on but no luck with any of the fields that I see in the props.conf that came with the app pre-defined, such as MID ICID etc.. Even when i use a query with MID defined using regex, I get the field now but no data.. I don't get it. here is my test query, note that the MID seen is phony but the one I use does exist. What I'm expecting is for the query to group all events with that MID. But something else is wrong as again I have the add-on installed but none of the fields are present for this index are present like MID, ICID, dcid, from, subject etc etc.

index=email sourcetype=/home/jm/2016-01-10.log | rex field=_raw "MID (?\d+)" | table mid 1234567

[javiergn]

Hi, your regex is wrong, try this instead:

index=email sourcetype=/home/jm/2016-01-10.log 
| rex field=_raw "MID (?<mid>\d+)" 
| table mid 1234567

With regards to the Cisco or Ironport app not working as expected I would suggest raising a new question as I don't have too much experience with those apps. Details such as your version of Splunk, Ironport, Apps, etc. are definitely needed. Examples of how your data looks like in Splunk and your raw data will help too. Asking too many things in one single question is normally a bad idea if you want to get useful answers quickly.

Thanks,
J

[jluo_splunk]

Hi Mendesjo, could you give examples of what the data looks like, and what search strings you have tried?