I have the following search which I would like to use to populate a summary index for reporting (run every 30 minutes or so to keep the summary index relatively up to date). Right now I am executing this transaction in saved searches to report on fields the cisco_esa_addon extracts (e.g. a search for a sender domain piped to "chart count(mailto) over mailfrom") but the searches take a long time to execute (up to 2 hours for a 24 hour period) as it operates on a very large set of data (~20 million events per day from our ironport cluster). The fillnull command is due to a csv lookup that is applied to the sourcetype cisco_esa that marks known large volume senders as "status=safe".
eventtype="cisco_esa" | transaction mid maxspan=180s | fillnull value=NULL status
A sample transaction is below:
<22>Sep 23 09:25:21 mail_logs: Info: Start MID 106908193 ICID 166122699
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 ICID 166122699 From: <
[email protected]>
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 ICID 166122699 RID 0 To: <
[email protected]>
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 ICID 166122699 RID 1 To: <
[email protected]>
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 ICID 166122699 RID 2 To: <
[email protected]>
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 ICID 166122699 RID 3 To: <
[email protected]>
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 ICID 166122699 RID 4 To: <
[email protected]>
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 Message-ID '<465C0721A14CC844B81CB20CA2E66025A269C1D97D@mchex2k7>'
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 Subject 'Question'
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 ready 4617 bytes from <
[email protected]>
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 matched all recipients for per-recipient policy Silas 2 in the inbound table
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 interim verdict using engine: CASE spam negative
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 using engine: CASE spam negative
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 interim AV verdict using Sophos CLEAN
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 antivirus negative
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 queued for delivery
<22>Sep 23 09:25:21 mail_logs: Info: Delivery start DCID 51145731 MID 106908193 to RID [0, 1, 2, 3, 4]
<22>Sep 23 09:25:21 mail_logs: Info: Message done DCID 51145731 MID 106908193 to RID [0, 1, 2, 3, 4]
<22>Sep 23 09:25:21 mail_logs: Info: MID 106908193 RID [0, 1, 2, 3, 4] Response 'Ok'
<22>Sep 23 09:25:21 mail_logs: Info: Message finished MID 106908193 done
I have tried following several of the suggestions posted in forum questions about populating a summary index from a transaction, but they either don't populate the summary index or they don't capture the original fields in such a fashion that I can report upon the data. Is there a way to preserve this as a multiline event with fields in a summary index?
... View more