I have data on which i want to apply 2 level of filtering before indexing
Let the complete data set be called A
Level1 : On complete data i want to keep only events that match some defined patterns ( regular expressions ) ( let the group of pattern be called as B )
So resulting pattern should be A - B
Level 2 : On the filtered events , i want to reject specified defined patterns and KEEP the remaining one ( Let the patterns be called C )
Result should be A - B - C
All this should happen during parsing stage only
see the docs about filter event data and send to queue for some good examples on this topic.
Create a nullQueue transform for B and a second for C and it should work for you.