topic Re: Events are not getting filtered using props.conf & transforms.conf in Getting Data In

Events are not getting filtered using props.conf & transforms.conf

sraji — Wed, 02 Sep 2020 04:04:59 GMT

I was wondering why all of the filters implemented are not working. Below is my props.conf & transforms.conf file

props.conf

[source::L:\\sample\\logs\\collections...*>]
TRANSFORMS-set= samplecollectionlogs

[source::L:\\sample\\logs\\(?:commands|webapps|partions)...*>]
TRANSFORMS-set1= samplecommandlogs

[source::L:\\sample\\logs\\engines...*>]
SEDCMD-maskfilterlist = s/\(\(not\(deniedlist1 in \('.*'\)\)\)\) /((not(deniedlist1 in ('_content_removed_by_splunk')))) /

transforms.conf

[samplecollectionlogs]
REGEX = (^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s(\{\d+\}\s)?(mapping|custom|TreePrefixBuilder|XB|ScdLookup|\s|\})|^[^0-9\]])
DEST_KEY = queue
FORMAT = indexQueue

[samplecommandlogs]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

Also my doubts is L:\\sample\\logs path are not defined in my heavy splunk(i.e where my props & transforms file reside) but these paths are defined in inputs of the universal forwarders. Source will also consider the monitor path from universal forwarders or should i define in heavy forwarder as well

Re: Events are not getting filtered using props.conf & transforms.conf

gcusello — Wed, 02 Sep 2020 06:34:48 GMT

Hi @sraji,

as you can see at https://docs.splunk.com/Documentation/Splunk/8.0.5/Forwarding/Routeandfilterdatad#Filter_event_data_and_send_to_queues to filter events keeping specific events and discarding the rest you have to put the command on the same row in props.conf.

You have only to put attention that the setnull must be before the other, something like this:

props.conf

[source::L:\\sample\\logs\\\\...*>] TRANSFORMS-set= samplecommandlogs,samplecollectionlogs [source::L:\\sample\\logs\\engines...*>] SEDCMD-maskfilterlist = s/\(\(not\(deniedlist1 in \('.*'\)\)\)\) /((not(deniedlist1 in ('_content_removed_by_splunk')))) /

transforms.conf:

[samplecollectionlogs] REGEX = (^\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2}\s(\{\d+\}\s)?(mapping|custom|TreePrefixBuilder|XB|ScdLookup|\s|\})|^[^0-9\]]) DEST_KEY = queue FORMAT = indexQueue [samplecommandlogs] REGEX = . DEST_KEY = queue FORMAT = nullQueue

Anyway, check the regexes in regex101 site.

If instead you want only to discard specific events, you can use only "samplecommandlogs".

Ciao.

Giuseppe

Re: Events are not getting filtered using props.conf & transforms.conf

sraji — Wed, 02 Sep 2020 07:03:52 GMT

Hi @gcusello ,

Yes i have made the changes as given but still the events are getting indexed from samplecommandlogs.

Re: Events are not getting filtered using props.conf & transforms.conf

sraji — Wed, 02 Sep 2020 07:06:40 GMT

Hi @gcusello , Yes i have made the changes as given but still the events are getting indexed from samplecommandlogs.

Re: Events are not getting filtered using props.conf & transforms.conf

gcusello — Wed, 02 Sep 2020 07:12:58 GMT

Hi @sraji,

where did you located the props.conf and transforms.conf? they must be located on Indexers or (when present) on Heavy Forwarders, not on Universal Forwarders.

Then, are you speaking of the first (keep specific events and discardithe rest) or the second (discard specific events)?

if the first, the meaning of the command is that you take only the events that match the regex and discard all the others.

If the second you directly discard the events that match the regex.

Did you restarted Splunk?

Ciao.

Giuseppe

Re: Events are not getting filtered using props.conf & transforms.conf

sraji — Wed, 02 Sep 2020 07:32:44 GMT

Hi @gcusello ,

Props.conf & transforms.conf are located under

<splunk_home>/etc/system/local/. Yes it is heavy forwarder because here only search & index is available.

for samplecollectionlogs i dont have any logs which are matching now so no events are filtered --> anyway i cant test this untill i have events which are related to this

for samplecommandlogs it needs to discard all the event matches --> this is not discarding the documents

Yes i have restarted the splunk instance from settings-->server controls--> restart splunk