topic Re: How do I exclude certain emails from being indexed? in Getting Data In

How do I exclude certain emails from being indexed?

capilarity — Thu, 14 May 2020 12:23:01 GMT

We have recently turned on journaling within MS Exchange which basically sends a copy of every item to a journaling mail box. We know the email address the process uses and this appears in the message tracking logs as additional emails - in short journaling has doubled our Splunk licence usage!!

We would like to exclude the journaling email address from being indexed.
Exchange can't turn off message tracking for certain email addresses

We are using UF on the Exchange servers with a load balanced intermediate level of Heavy forwarders.

We have tried to apply the exclusion based on this answer - Answer 289736 - how to exclude a sourcetype from being indexed, but using a regex that picks up the journaling email address.

The address we want to exclude is: journal@ev.local

We've tried it on the Intermediate Heavy forwarders and on the index servers with no effect.

The config we have applied is:

props.conf
[MSExchange:2013:MessageTracking]
TRANSFORMS-JournalRemoval = JournalRemoval

transforms.conf
[JournalRemoval]
REGEX =.*journal@ev.*
DEST_KEY = queue
FORMAT = nullqueue

Any ideas why this might not be working?
Thanks

Re: How do I exclude certain emails from being indexed?

gcusello — Thu, 14 May 2020 12:52:18 GMT

Hi @capilarity,
filter must be on Indexers if there isn't any Heavy Forwarders, otherwise on HFs.
the instruction to filter data are at https://docs.splunk.com/Documentation/Splunk/8.0.3/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

Anyway, in transforms.conf try one or the other of these regexes:

REGEX=journal@ev
REGEX=(journal@ev)

Ciao.
Giuseppe

Re: How do I exclude certain emails from being indexed?

capilarity — Fri, 15 May 2020 15:13:16 GMT

Thanks Giuseppe, I had found the docs you highlighted but they are about including the ones that match the regex and discard the rest, we are trying to do the other way round, discard those that match, index the rest.

I've also tried the different combinations of the regex you suggested on both the intermediate HF and the indexer.
Sill no luck 😞

Re: How do I exclude certain emails from being indexed?

gcusello — Fri, 15 May 2020 15:54:44 GMT

Hi @capilarity,
sorry few lines above:
https://docs.splunk.com/Documentation/Splunk/8.0.3/Forwarding/Routeandfilterdatad#Discard_specific_events_and_keep_the_rest

Ciao.
Giuseppe