topic Re: SEDCMD log filtering regex needed in Getting Data In

How to achieve regex to keep the highlighted parts from the below raw log and remove the rest using SEDCMD?

DanAlexander — Fri, 16 Jun 2023 13:59:45 GMT

Hello community,

I am looking for a regex to keep the highlighted parts from the below raw log and remove the rest using SEDCMD

c-ip=XXX.XXX.XXX.XXX rs-Content-Type="application/javascript" cs-auth-groups="xxxxxx\ROLE.STD.MSTeams" cs-bytes=888 cs-categories="Technology/Internet;NetSkope_XXX" cs-host=xxxxxxxx cs-ip=XXX.XXX.XXX.XXX cs-method=GET cs-uri-port=443 cs-uri-scheme=https cs-User-Agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Teams/1.6.00.12455 Chrome/ XXX.XXX.XXX.XXX Electron/XX.1.8 Safari/5XX.3X" cs-username=XXXX dnslookup-time=0 duration=0 rs-status=200 rs-version=HTTP/1.1 s-action=TCP_HIT s-ip=XXX.XXX.XXX.XXX service.name="XXXXX HTTP" service.group="Standard" s-supplier-ip=XXX.XXX.XXX.XXX s-supplier-name=XXX.XXX.XXX.XXX sc-bytes=1XXX711 sc-filter-result=OBSERVED sc-status=200 time-taken=31 c-url="/xxxxxxxx.net/midgard/versionless/livepersonacardstrings_f8axxxad2fc4867bf1300xxxx06c7057c23.js" cs-Referer="httpsxxx.com/" cs-auth-groups="ccccccc\ROLE.STD.MSTeams" cs-headerlength=667 cs-threat-risk=2 r-ip=XXX.XXX.XXX.XXX s-connect-type=Unknown s-icap-status=ICAP_NOT_SCANNED s-sitename=https.forward-proxy s-source-port=0 s-supplier-country="None" sr-Accept-Encoding=gzip,%20deflate,%20br,%20identity x-auth-credential-type=NTLM x-cookie-date=Thu,%2015-Jun-23%2009:15:15%20GMT x-cs-connection-negotiated-cipher=XXXX_256_GCM_SHA384 x-cs-connection-negotiated-cipher-size=256 x-cs-connection-negotiated-ssl-version=TLSv1.3 x-cs-Referer-uri=https://teams.microsoft.com/ x-cs-Referer-uri-address=XXX.XXX.XXX.XXX x-cs-Referer-uri-host=teams.microsoft.com x-cs-Referer-uri-hostname=teams.microsoft.com x-cs-Referer-uri-port=XXX x-cs-Referer-uri-scheme=https x-cs-Referer-uri-stem=https://teams.microsoft.com/ x-exception-sourceline=0 x-rs-certificate-hostnamexxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxertificate-observed-errors=none x-rs-certificate-xxxxxxxxxxxxxxxnet" x-rs-certificate-validate-status=CERT_VALID x-rs-connection-negotiated-cipher=TLS_AES_256_GCM_SHA384 x-rs-connection-negotiated-cipher-size=256 x-rs-connection-negotiated-ssl-version=TLSv1.3 cs-uri-extension=js cs-uri-path=/midgard/versionless/livepersonacardstrings_f8aa070xxxxxxxxx4867bf13000eac47f306c7057c23.js c-uri-pathquery=/midgard/versionless/livepersonacardstrings_f8aa070ad2fc4867bf1300xxxxxxxxxxxc7057c23.js

Thank you!

Re: SEDCMD log filtering regex needed

ITWhisperer — Thu, 15 Jun 2023 11:19:11 GMT

| rex mode=sed "s/(?:^.*)(?<cip>c-ip=\S+)\s.*(?<csbytes>cs-bytes=\S+)\s.*(?<csip>cs-ip=\S+)\s.*(?<csmethod>cs-method=\S+)\s.*(?<csusername>cs-username=\S+)\s.*(?<sip>s-ip=\S+)\s.*(?<ssupplierip>s-supplier-ip=\S+)\s.*(?<csreferer>cs-Referer=\S+)\s.*(?<csauthgroups>cs-auth-groups=\S+)\s.*(?<csthreatrisk>cs-threat-risk=\S+)\s.*(?<rip>r-ip=\S+)\s.*(?<xcsrefererurihostname>x-cs-Referer-uri-hostname=\S+)\s.*/\1 \2 \3 \4 \5 \6 \7 \8 \9 \10 \11 \12/g"

Re: SEDCMD log filtering regex needed

DanAlexander — Thu, 15 Jun 2023 15:21:04 GMT

Hi @ITWhisperer,

Awesome as always. Worked first time.

I wanted to ask would you be willing to help me out with the following, please?

I need help reducing Events containing 4688 and ParentProcessName=*splunkd.exe

There is an excerpt from the log:

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid=' XXXXXXXX -4994-a5ba-3e3b0328c30d}'/><EventID>4688</EventID><Version>2</Version><Level>0</Level><Task>13312</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2023-06-13T10:39:41.797279900Z'/><EventRecordID>12536409</EventRecordID><Correlation/><Execution ProcessID='4' ThreadID='15216'/><Channel>Security</Channel><Computer> XXXXXXXX </Computer><Security/></System><EventData><Data Name='SubjectUserSid'>S-1-5-18</Data><Data Name='SubjectUserName'>XXXXXXXX</Data><Data Name='SubjectDomainName'> XXXXXXXX </Data><Data Name='SubjectLogonId'>0 XXXXXXXX 7</Data><Data Name='NewProcessId'>0x2734</Data><Data Name='NewProcessName'>C:\Program Files\SplunkUniversalForwarder\bin\splunk-MonitorNoHandle.exe</Data><Data Name='TokenElevationType'>%%1936</Data><Data Name='ProcessId'>0x17d4</Data><Data Name='CommandLine'></Data><Data Name='TargetUserSid'>S-1-0-0</Data><Data Name='TargetUserName'>-</Data><Data Name='TargetDomainName'>-</Data><Data Name='TargetLogonId'>0x0</Data><Data Name='ParentProcessName'>C:\Program Files\SplunkUniversalForwarder\bin\splunkd.exe</Data><Data Name='MandatoryLabel'>XXXXXXXX -16384</Data></EventData></Event>

Can anyone help me create the appropriate regex I can use within the SEDCMD?

After the reduction the above event the result I am after should look something like this: <EventID>4688</EventID><Data Name='ParentProcessName'>C:\Program Files\SplunkUniversalForwarder\bin\splunkd.exe</Data>

Is the stanza I need to place this looks like this: [WinEventLog]

Thank you!

Re: SEDCMD log filtering regex needed

ITWhisperer — Wed, 21 Jun 2023 08:55:49 GMT

Here's how to do it in SPL:

| makeresults | fields - _time | eval _raw=" <Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid=' XXXXXXXX -4994-a5ba-3e3b0328c30d}'/><EventID>4688</EventID><Version>2</Version><Level>0</Level><Task>13312</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2023-06-13T10:39:41.797279900Z'/><EventRecordID>12536409</EventRecordID><Correlation/><Execution ProcessID='4' ThreadID='15216'/><Channel>Security</Channel><Computer> XXXXXXXX </Computer><Security/></System><EventData><Data Name='SubjectUserSid'>S-1-5-18</Data><Data Name='SubjectUserName'>XXXXXXXX</Data><Data Name='SubjectDomainName'> XXXXXXXX </Data><Data Name='SubjectLogonId'>0 XXXXXXXX 7</Data><Data Name='NewProcessId'>0x2734</Data><Data Name='NewProcessName'>C:\Program Files\SplunkUniversalForwarder\bin\splunk-MonitorNoHandle.exe</Data><Data Name='TokenElevationType'>%%1936</Data><Data Name='ProcessId'>0x17d4</Data><Data Name='CommandLine'></Data><Data Name='TargetUserSid'>S-1-0-0</Data><Data Name='TargetUserName'>-</Data><Data Name='TargetDomainName'>-</Data><Data Name='TargetLogonId'>0x0</Data><Data Name='ParentProcessName'>C:\Program Files\SplunkUniversalForwarder\bin\splunkd.exe</Data><Data Name='MandatoryLabel'>XXXXXXXX -16384</Data></EventData></Event>" | rex mode=sed "s/.*(?<eventId><EventID>4688<\/EventID>).*(?<parentProcess><Data Name='ParentProcessName'>C:\\\\Program Files\\\\SplunkUniversalForwarder\\\\bin\\\\splunkd.exe<\/Data>).*/\1\2/g"

For SEDCMD, you might need fewer backslashes:

s/.*(?<eventId><EventID>4688<\/EventID>).*(?<parentProcess><Data Name='ParentProcessName'>C:\\Program Files\\SplunkUniversalForwarder\\bin\\splunkd.exe<\/Data>).*/\1\2/g

Re: SEDCMD log filtering regex needed

DanAlexander — Wed, 21 Jun 2023 09:11:42 GMT

Thank you for replying back @ITWhisperer Much appreciated!

I do struggle at the moment with the sourcetype/source within the props.conf

I did put lots of entries but not sure why this still not reducing the logs

I used:

[WinEventLog]

SEDCMD=regex_here

[wineventlog]

SEDCMD=s/.*/ParentProcessName/g

[xmlwineventlog]

SEDCMD=regex_here

[XmlWinEventLog]

SEDCMD=regex_here

[source::WinEventLog:Security]

SEDCMD=regex_here

[WinEventLog:Security]

SEDCMD=regex_here

[WinEventLog:ForwardedEvents]

SEDCMD=regex_here

[source::WinEventLog:ForwardedEvents]

SEDCMD=regex_here

---------------------------

I am not sure what to use within the squared brackets to make this work.

Regards,

Dan