Filtering Data by props.conf and transform.conf

[marco_massari11]

Hi,

I need to filter out some events from a syslog source. The events  are like this:

Apr 28 14:15:09 10.130.4.203 Apr 28 14:15:09 hostname: User ****  : Sign Off, ID: **, InstID: 4731, IPAddress: *****, FolderID: 0, Username: ******, AgentBrand: -, AgentVersion: -, XFerSize: 0, Error: 0

Apr 28 14:15:09 10.130.4.203 Apr 28 14:15:09 hostname: User ****  : Upload, ID: **, InstID: 4731, IPAddress: *****, FolderID: 1234, Username: ******, AgentBrand: -, AgentVersion: -, XFerSize: 0, Error: 0

Apr 28 14:15:09 10.130.4.203 Apr 28 14:15:09 hostname: User ****  : Sign Off, ID: **, InstID: 2819, IPAddress: *****, FolderID: 0, Username: ******, AgentBrand: -, AgentVersion: -, XFerSize: 0, Error: 0

I have two different InstID (4731 and 2819) and many FolderID, so  I need to keep all the events with InstID:2189 and the events whith InstID:4731 and FolderID:0, so my goal is to discard by props.conf and transforms.conf all the events that have InstID:4731 and FolderID different from 0

Any help? Thanks in advance

[maciep]

I would say you have two main choices.  Either, try to come up with a regex to match those that you want to discard.  Or come up with the regex'es for those that you want to keep.  

For the former, you would use use that regex to send to the nullQueue.  For the latter, you would set all events to the nullQueue, then set the events you want to keep to the indexQueue. 

There are plenty of examples out here and in the docs on how to do that.  For the regex, here is an example of the former that may work for you - white space is assumed to be consistent and the same as your examples.  Obviously would need to be tested, but this is attempting to match 4731 events with any FolderID that isn't a single 0.

InstID:\s*4731,.*?FolderID:\s+(0[^,]|[^0])