Keep specific events and discard the rest

[yosoypako]

Hello
I want to index the events in the firewalls log based in the alert level and the virtual domain in witch they have been generated. I have followed the guide in https://docs.splunk.com/Documentation/Splunk/7.0.2/Forwarding/Routeandfilterdatad#Perform_selective_...
but the indexer is indexing all the events not only the events that match the regular expression. I think that the setnull stanza may not be working. But I am not sure how to fix it

-This is the contain in props.conf

[source::/opt/LOGs/firewalls]
TRANSFORMS-set= setnull,setindexone,setindextwo,setindexthree,setindexfour

-And this is the contains in transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setindexone]
REGEX = level="(error|critical|alert|emergency)".*(vd="one")
DEST_KEY = queue
FORMAT = indexQueue

[setindextwo]
REGEX = level="(critical|alert|emergency)".*(vd="two")
DEST_KEY = queue
FORMAT = indexQueue

[setindexthree]
REGEX = level="(critical|alert|emergency)".*(vd="three")
DEST_KEY = queue
FORMAT = indexQueue

[setindexfour]
REGEX = level="(alert|emergency)".*(vd="four")
DEST_KEY = queue
FORMAT = indexQueue

-Also, I have checked with the command "btool --app=Splunk_TA_fortinet_fortigate transforms list and props list" that the .conf files configuration is being loaded.

Thanks

[yosoypako]

Ok, now it is working. I have added the props.conf and tranforms.conf to the forwarder (/opt/splunk/etc/system/local/) and it is working.
For future similar issues: How can I check if the input data is being already parsed by the forwarder before reaching the indexer? As I have read if the indexer receives the data already parsed by the forwarders it will not transform and filter it and will be directly indexed

Thanks for your help.

[FrankVl]

If that forwarder is a universal forwarder as you claimed, it doesn't make too much sense that this fixes it...

[richgalloway]

@yosoypako If your problem is resolved, please accept the answer to help future readers.

[yosoypako]

Hello.

If i use the btool command (splunk.exe cmd btool props list fgt_utm --debug) I can see that the setnull stanza that is being chosen is this one, with REGEX = .

Thanks.

[FrankVl]

How can you see that from a btool on props?

Can you run a btool on transforms and grep it for setnull, to see if there are conflicting setnull stanzas?

[lakshman239]

Are you trying to send events to diff indexes based on the values of vd? If so, you would need DEST_KEY = _MetaData:Index and FORMAT = yourindexname

[yosoypako]

Hello

I want to send all the logs from the different vd to the same index but I want to filter witch alert level messages are going to be indexed from each vd. So in one index I could index only the alert and emergency level but in other vd the critical, alert nad emergency levels.
I have not written this in the initial post but there are two different splunk machine in this set up an indexer and a universal forwarder. I am editing the props.conf and transforms.conf files only on the indexer, not on the forwarder. How can I confirm that the forwarder is not parsing any event?

Thanks.

[FrankVl]

If there is only a UF and an indexer involved, no heavy forwarder, then the indexer should indeed be the right place for this config.

"setnull" is a very generic name though. Any chance you have another transforms stanza named that way, that happens to take precedence and has a more specific REGEX setting, that doesn't match your firewall data?

[yosoypako]

Hello
I have tried to use the sourcetype in the props.conf file. But it is still not working.
-If I use the btool *cmd btool props list * it shows:

TRANSFORMS =
TRANSFORMS-set = setnull,setindexone,setindextwo,setindexthree,setindexfour

-And If I use the btool command with both the app and the sourcetype it shows:

TRANSFORMS-set = setnull,setindexone,setindextwo,setindexthree,setindexfour

Any ideas why it is not filtering the events

Regards.