topic Re: Help filter out unwanted data from indexing using nullqueue Please in Getting Data In

Help filter out unwanted data from indexing using nullqueue Please

marcusmartin — Fri, 12 Feb 2021 09:00:22 GMT

Hi if someone could please help that would be great, I have events showing up in the indexer that are pushing me over my license, alot of it is useless to me information and i have been trying to wrap my head around filtering it out using regex but i just cant get my head around it.

Below is a typical event i would like to rid my indexer of, i cant just block all the events with 4634 as some of them are valid, but i would like to block all events where the "Targetusersid" is similar to DOMAIN\ABC-12345$

Can anyone help

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-A5BA-3E3B0328C30D}'/><EventID>4634</EventID><Version>0</Version><Level>0</Level><Task>12545</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2021-02-12T08:24:29.977950700Z'/><EventRecordID>314243098</EventRecordID><Correlation/><Execution ProcessID='852' ThreadID='12388'/><Channel>Security</Channel><Computer>domaincontoller.domainname</Computer><Security/></System><EventData><Data Name='TargetUserSid'>DomainName\machine-name$</Data><Data Name='TargetUserName'>Machine-Name$</Data><Data Name='TargetDomainName'>DomainName</Data><Data Name='TargetLogonId'>0x22b9251d</Data><Data Name='LogonType'>3</Data></EventData></Event>

Props.conf

[XmlWinEventLog:Security]
TRANSFORMS-xml = xmlnull
REGEX=(?m)^EventCode="(4662|566)" Message="Object Type:(?!\s*groupPolicyContainer)"
REGEX=(?m)^EventCode="(4656|4670|4663|4703|4658|4688)" Message="Account Name:(\W+\w+$)"
REGEX=(?m)^EventCode="(4688|4689)" Message="%SplunkUniversalForwarder%"
REGEX=(?m)^EventCode="6278" Message="Network Policy Server granted full access to a user because the host met the defined health policy."
REGEX=(?m)^EventCode="(4624|4634|4627|4648)" Message="Account\sName:.*[\S\s]*Account\sName:\s+[\S+]+[\$]"

Transforms.conf

[xmlnull]
REGEX= NO idea 😞
DEST_KEY = queue
FORMAT = nullQueue

Re: Help filter out unwanted data from indexing using nullqueue Please

scelikok — Fri, 12 Feb 2021 09:57:25 GMT

Hi @marcusmartin,

Since you are using XML format you can filter events using $XmlRegex directive. There is no need to use nullQueue. Also the REGEX settings in props does not work, you should put them into inputs.conf on universal forwarder changing to XML format.

You can use below for filtering

inputs.conf [WinEventLog://Security] blacklist2 = $XmlRegex="<EventID>4634<\/EventID>.*<Data Name='TargetUserSid'>[^\\]+\\[^<]+\$"

Please try below;

Please also look at below document for filtering unnecessary data from Windows logs. You can use this filters also.

https://docs.splunk.com/Documentation/WindowsAddOn/8.1.1/User/Configuration#Configure_event_cleanup_best_practices_in_props.conf

Re: Help filter out unwanted data from indexing using nullqueue Please

marcusmartin — Fri, 12 Feb 2021 10:52:42 GMT

Perfect response, thankyou so much i would never have figured it out im sure. much respect.

Re: Help filter out unwanted data from indexing using nullqueue Please

marcusmartin — Fri, 12 Feb 2021 12:06:31 GMT

Could I be a real pain How would you block TargetUserSid'>NT AUTHORITY\SYSTEM using regex?

Re: Help filter out unwanted data from indexing using nullqueue Please

marcusmartin — Fri, 12 Feb 2021 12:08:10 GMT

blacklist2 = $XmlRegex="<EventID>4634<\/EventID>.*<Data Name='TargetUserSid'>NT AUTHORITY\SYSTEM"

I just tried this but im thick when it comes to regex

Re: Help filter out unwanted data from indexing using nullqueue Please

scelikok — Fri, 12 Feb 2021 13:53:19 GMT

Hi @marcusmartin,

You can try below; (You should escape "\")

blacklist2 = $XmlRegex="<EventID>4634<\/EventID>.*<Data Name='TargetUserSid'>NT AUTHORITY\\SYSTEM"

Re: Help filter out unwanted data from indexing using nullqueue Please

marcusmartin — Fri, 12 Feb 2021 15:22:57 GMT

blacklist5 = $XmlRegex="NT AUTHORITY\\SYSTEM"

I used this in the end as i really wasnt bothered what event id it came from im not intrested in events from that particular system account. Seems to be working and yes you are right i just learned today that you have to escape any \ and put \\ for the regex to handle it.

thanks again for your time its very much appreciated.

Cuppa coffee to you.

Regards