topic How to filter XmlWinEventLog in Heavy Forwarder with regex? in Splunk Search

How to filter XmlWinEventLog in Heavy Forwarder with regex?

borshoff — Mon, 13 Mar 2017 14:45:39 GMT

Hi,
I have XML rendered log from sysmon and i need to extract from this log only interesting fields, for example:

Image|UtcTime|ProcessGuid|CommandLine|User|ParentProcessGuid|ParentImage|ParentCommandLine|Hashes

But my conf doesn't work.
What i did wrong and how to fix that?

here is the sample xml

- <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
- <System>
  <Provider Name="Microsoft-Windows-Sysmon" Guid="{5770385F-C22A-43E0-BF4C-06F5698FFBD9}" /> 
  <EventID>1</EventID> 
  <Version>5</Version> 
  <Level>4</Level> 
  <Task>1</Task> 
  <Opcode>0</Opcode> 
  <Keywords>0x8000000000000000</Keywords> 
  <TimeCreated SystemTime="2017-03-13T12:16:18.234566900Z" /> 
  <EventRecordID>1098206</EventRecordID> 
  <Correlation /> 
  <Execution ProcessID="2416" ThreadID="2476" /> 
  <Channel>Microsoft-Windows-Sysmon/Operational</Channel> 
  <Computer>HOSTNAME</Computer> 
  <Security UserID="S-1-5-18" /> 
  </System>
- <EventData>
  <Data Name="UtcTime">2017-03-13 12:16:18.203</Data> 
  <Data Name="ProcessGuid">{EF92ED9B-8D92-58C6-0000-0010B2A27B04}</Data> 
  <Data Name="ProcessId">2832</Data> 
  <Data Name="Image">C:\Windows\System32\cmd.exe</Data> 
  <Data Name="CommandLine">"C:\Windows\system32\cmd.exe" /c type "C:\ProgramData\****.txt"</Data> 
  <Data Name="CurrentDirectory">c:\program files\*****\</Data> 
  <Data Name="User">NT AUTHORITY\SYSTEM</Data> 
  <Data Name="LogonGuid">{****************************}</Data> 
  <Data Name="LogonId">0x3e7</Data> 
  <Data Name="TerminalSessionId">0</Data> 
  <Data Name="IntegrityLevel">System</Data> 
  <Data Name="Hashes">SHA1=0F3C4FF28F354AEDE2,MD5=5746BD7E255DD61,SHA256=DB06C3534964E3FC79D0CA336F4A0FE724B75AAFF386,IMPHASH=D00585440EB0A</Data> 
  <Data Name="ParentProcessGuid">{**************************}</Data> 
  <Data Name="ParentProcessId">1564</Data> 
  <Data Name="ParentImage">C:\Program Files\****.exe</Data> 
  <Data Name="ParentCommandLine">"C:\Program Files\******" 1452</Data> 
  </EventData>
+ <RenderingInfo Culture="en-US">
  <Message> **************************************************************</Message> 
  <Level>Information</Level> 
  <Task>Process Create (rule: ProcessCreate)</Task> 
  <Opcode>Info</Opcode> 
  <Channel /> 
  <Provider /> 
  <Keywords /> 
  </RenderingInfo>
  </Event>

And this is my conf:

inputs.conf

[WinEventLog://ForwardedEvents]
    disabled = false
    start_from = oldest
    current_only = 0
    checkpointInterval = 5
    renderXml = true
    suppress_text = 1
    index = sysmon
    sourcetype=XmlWinEventLog:Microsoft-Windows-Sysmon/Operational
    whitelist1 = 1,5,6

props.conf

 [source::WinEventLog://ForwardedEvents]
    TRANSFORMS-setnull = sysmon-setnull
    TRANSFORMS-keep = sysmon-keep

transforms.conf

[sysmon-setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[sysmon-keep]
REGEX = (?i)Name=".*(Image|UtcTime|ProcessGuid|CommandLine|User|ParentProcessGuid|ParentImage|ParentCommandLine|Hashes)"
DEST_KEY = queue
FORMAT = indexQueue

Re: How to filter XmlWinEventLog in Heavy Forwarder with regex?

gcusello — Mon, 13 Mar 2017 15:16:45 GMT

Hi borshoff,
you cannot filter your events to take only a part of them (only selected fields), You can filter events to take (or discard) all (full) events that match a regex, the only way to limit the dimensions of your events is to put a limit to the number of characters to take for each event (see limits.conf).
Bye.
Giuseppe

Re: How to filter XmlWinEventLog in Heavy Forwarder with regex?

somesoni2 — Mon, 13 Mar 2017 15:31:04 GMT

The configuration that you've is for event filtering, means if an (whole) event is matching a regex, drop the event altogether. The configuration that you're looking for is data masking where you can replace all the not-required lines with blank.

I'm guessing that your xml <Event> has section <EventData> and that's the only thing you want to ingest and drop everything else. So give this a try (inputs.conf can stay the same)

props.conf (on indexer/heavy forwarder)

[source::WinEventLog://ForwardedEvents]
SHOULD_LINEMERGE = false
LINE_BREAKER = ([\r\n]+)(?=\s*\<EventData\>)
SEDCMD-removeheader=s/^(\s*\<Event xmlns(.+[\r\n]*)+)//
SEDCMD-removefooter=s/(\s*\<RenderingInfo (.+[\r\n]*)+)//

Re: How to filter XmlWinEventLog in Heavy Forwarder with regex?

woodcock — Mon, 13 Mar 2017 19:19:37 GMT

The way to do this is to use SEDCMD to replace the undesired parts with nothing:

https://docs.splunk.com/Documentation/Splunk/latest/Data/Anonymizedata

Re: How to filter XmlWinEventLog in Heavy Forwarder with regex?

borshoff — Tue, 14 Mar 2017 16:26:47 GMT

Hi, somesoni2
Yes you right, I need only EventData section.
I've try props.conf on heavy forwarder as you describe, but it doesnt work. Headers and footers don't remove.

Re: How to filter XmlWinEventLog in Heavy Forwarder with regex?

somesoni2 — Tue, 14 Mar 2017 16:40:58 GMT

I've updated the regex for removefooter.

How are you getting the data into heavy forwarder, from universal forwarder? Were heavy forwarder restarted after making the change? Try keep these configurations on Universal forwarder.

Re: How to filter XmlWinEventLog in Heavy Forwarder with regex?

borshoff — Thu, 16 Mar 2017 13:11:36 GMT

I found what was wrong.
In input.conf we set "renderXml = true" . That's why props.conf doesn't apply to source::WinEventLog://ForwardedEvents. Cause source::WinEventLog://ForwardedEvents doesn't exist !

When i change it to "renderXml = false", filter start working!
But, i still need get this events in XML. Is there any way to do that?

"How are you getting the data into heavy forwarder, from universal forwarder?"
No, we collect all events by Windows Event collecor server in ForwardedEvents log. On the same VM we deploy heavy forwarder + Windows_TA addon with all necessary conf.