Is it possible to do a groupby operation on multiline Cisco Ironport before indexing?

[ananthkumar12]

I am trying to do a groupby operation at index time on Ironport logs. I have looked in all the documents and posts and they all talk about search time groupby using stats.

Scenario: We are getting Ironport Mail logs. The logs are pretty chatty and multiline. If that alone was the case, we could have done props line-break statements and be done with it. These logs, however, are Multiline as well as Multi-message (parallel: shown below). I want to try separate the logs belonging to individual email and group them before they hit indexers. Any help will be very appreciated.

Scenario: Attached
As you can see there is not specific start and end strings to the log so it is hard to do MUST_BREAK_BEFORE or MUST_NOT_BREAK_BEFORE regexs

[mikaelbje · Accepted solution]

You won't be able to do this. AFAIK there is also no guarantee that a new mail conversation won't show up in the middle of that conversation, so you can't really frame it. What you can do is work with this after indexing and feed it to a summary index. I have an app that does this which I intend to release if you want to test.

[ananthkumar12]

Thanks @mikaelbje
Can you please send me the app. i will see if it does the trick.
Cheers

[mikaelbje]

Hi!

You can pull the add-on from https://github.com/inspired/TA-cisco-esa-extras

It groups email transactions into a summary index once an hour. Copy default/macros.conf to local/macros.conf and modify the file to change the index where you want your summary data stored. The default in the app is "esa_summary", but you can replace that with "summary" to use the built-in summary index in Splunk.

When you get it to work you can tweak the run interval of the saved search.

Please note that this add-on requires that you also have the Splunk_TA_cisco-esa add-on installed and that your data goes into the right sourcetype as defined by that add-on.

[ananthkumar12]

Thank you Very much. i will pull it tonight.