topic Re: Keep specific events and discard the rest in Getting Data In

Keep specific events and discard the rest

Michael_Schyma1 — Wed, 14 Nov 2012 17:30:14 GMT

Yes i have seen the documentation and i am having probelm getting my stanza's to work. I just want to grab Directory Administrators and Master Web Resource Admins and get rid of the rest of the messages since we will not be doing anything with them.

Here is props.conf:

[source::/n01/data/bsm/hand/access.20121113-082934]
TRANSFORMS-set= setnullldap,setparsingldap

Here is transforms.conf:

[setnullldap]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setparsingldap]
REGEX = Directory Administrators|Master Web Resource Admins
DEST_KEY = queue
FORMAT = indexQueue

SAMPLE OF RAW DATA THAT I WANT TO KEEP BECAUSE IT INCLUDES cn= Directory Administrators OR cn= Master Web Resource Admins. I want to discard the rest of the events besides the ones that have those two admins as CN's.

SAMPLE DATA BELOW:

[13/Nov/2012:09:00:04 -0500] conn=6333991 op=163 SRCH base="cn=Master Web Resource Admins,obapp=PSC,o=Oblix,o=test.com" scope=0 filter="(obuniquememberStr=uid=appcdt2,ou=people,ou=intranet,dc=test,dc=com)" attrs="1.1"

[13/Nov/2012:09:00:05 -0500] conn=6333969 op=443 SRCH base="cn=Directory Administrators,o=Oblix,o=test" scope=0 filter="(obuniquememberStr=uid=appcdt2,ou=people,ou=intranet,dc=test,dc=com)" attrs="1.1"

If anymore information is needed please just post. Thank you guys so much.

Re: Keep specific events and discard the rest

lguinn2 — Wed, 14 Nov 2012 18:12:53 GMT

Your regex is broken. I think this may work:

REGEX =(?m) (?:Directory\sAdministrators)|(?:Master\sWeb\sResource\sAdmins)

Also, do you want to apply this to only a single file, or to all data of that sourcetype? To apply the transformation to only a single file seems unusual. It would be more typical to see

[yoursourcetypehere]
TRANSFORMS-set= setnullldap,setparsingldap

Re: Keep specific events and discard the rest

Michael_Schyma1 — Wed, 14 Nov 2012 19:10:43 GMT

Hm i still did not get the results that i was looking for. I will post two samples of the raw data that i am looking at. The raw data is the data i want to keep because it has cn=Directory Administrators and cn=Master Web Resource Admins. Everything else i would like to discard.

Re: Keep specific events and discard the rest

bharathi86 — Mon, 26 Nov 2012 10:42:51 GMT

Try the below setup.
Hope this work for you.

[setnullldap]
REGEX = [\d+\/\w+\/\d+:\d+:\d+:\d+\s+-\d+]
DEST_KEY = queue
FORMAT = nullQueue

[setparsingldap]
REGEX = [\d+\/\w+\/\d+:\d+:\d+:\d+\s+-\d+].*(\Master\sWeb\sResource|\Directory\sAdministrators)
DEST_KEY = queue
FORMAT = indexQueue

[source::/n01/data/bsm/hand/access.20121113-082934]
Linebreaking =[\d+\/\w+\/\d+:\d+:\d+:\d+\s+-\d+]
TRANSFORMS-set= setnullldap,setparsingldap

I did the similar setup recently, it worked fine for me.
If need more information .

Re: Keep specific events and discard the rest

lguinn2 — Mon, 26 Nov 2012 21:57:43 GMT

I have updated my previous answer. I don't think the other answer will work for you, as there are spaces in your data...