Convert ingested WinEventLog in JSON format

[SplunkExplorer]

Hi Splunkers, I have a request by my customer.

We have, like in many prod environments, Windows logs. We know that we can see events on Splunk Console, with Splunk Add-on for Microsoft Windows , in 2 way: Legacy format (like  the original ones on AD) or XML. Is it possible to see them on JSON format? If yes, we can achieve this directly with above addon or we need other tools?

[SplunkExplorer]

Thanks all you guys, you gave me a lot of hints and you have been very helpfull

[tscroggins]

Hi @SplunkExplorer,

I had a bit of fun with this today.

Splunk ingests Windows event log events using a modular input, splunk-winevtlog.exe, which stream events in xml mode to stdout. The event stream looks like this:

<stream>
<event stanza="WinEventLog://...">
<time>...</time>
<data>...</data>
<source>...</source>
<sourcetype>...</source>
<index>...</index>
</event>
<event>
...
</event>
<event>
...
</event>
...
</stream>

The schema and output behavior are documented at https://dev.splunk.com/enterprise/docs/developapps/manageknowledge/custominputs/modinputsscript/#XML....

The startup path for splunk-winevtlog.exe is stored in %SPLUNK_HOME%\bin\scripts\splunk-winevt.log.path.

With knowledge of these two things in hand, we can begin the work of writing a wrapper for splunk-winevtlog.exe that transforms the output of the command.

I've written the first interation of a PowerShell 5.1 / .NET Framework 4.0 script to read the output of splunk-winevtlog.exe, modify the data and sourcetype elements, and write the new stream to stdout.

Before writing the script, though, the translation from WinEventLog and XmlWinEventLog to JSON must be defined.

In the case of WinEventLog, it's fairly straightforward to skip the timestamp and convert the key-value pairs into JSON keys. The following input (truncated for brevity):

12/03/2023 08:28:32 PM
LogName=Security
EventCode=4688
EventType=0
ComputerName=host1
SourceName=Microsoft Windows security auditing.
Type=Information
RecordNumber=123
Keywords=Audit Success
TaskCategory=Process Creation
OpCode=Info
Message=A new process has been created.

Creator Subject:
	Security ID:		host1\user

becomes:

{"LogName":"Security","EventCode":"4688","EventType":"0","ComputerName":"host1","SourceName":"Microsoft Windows security auditing.","Type":"Information","RecordNumber":"123","Keywords":"Audit Success","TaskCategory":"Process Creation","OpCode":"Info","Message":"A new process has been created.\nCreator Subject:\n\tSecurity ID:\t\thost1\\user"}

In the case of XmlWinEventLog, we need to choose an object format, as compound elements, element values, and attributes don't translate directly arrays and keys. For the example, I handle EventData as an array, attributes as keys, and element values as a key named Value. Empty elements, <Foo/> are dropped. Elements with empty values, <Foo></Foo>, are retained. The following input (truncated for brevity):

<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-Security-Auditing' Guid='{54849625-5478-4994-a5ba-3e3b0328c30d}'/><EventID>4688</EventID><Version>2</Version><Level>0</Level><Task>13312</Task><Opcode>0</Opcode><Keywords>0x8020000000000000</Keywords><TimeCreated SystemTime='2023-12-04T00:28:32.0000000Z'/><EventRecordID>1234</EventRecordID><Correlation/><Execution ProcessID='1234' ThreadID='1234'/><Channel>Security</Channel><Computer>host1</Computer><Security/></System><EventData><Data Name='SubjectUserSid'>host1\user</Data></EventData></Event>

becomes:

{"Event":{"System":{"Provider":{"Name":"Microsoft-Windows-Security-Auditing","Guid":"{54849625-5478-4994-a5ba-3e3b0328c30d}"},"EventID":{"Value":"4688"},"Version":{"Value":"2"},"Level":{"Value":"0"},"Task":{"Value":"13312"},"Opcode":{"Value":"0"},"Keywords":{"Value":"0x8020000000000000"},"TimeCreated":{"SystemTime":"2023-12-04T00:28:32.0000000Z"},"EventRecordID":{"Value":"123"},"Execution":{"ProcessID":"1234","ThreadID":"1234"},"Channel":{"Value":"Security"},"Computer":{"Value":"host1"}},"EventData":[{"Name":"SubjectUserSid","Value":"host1\\user"}]}}

%SPLUNK_HOME%\bin\scripts\splunk-winevt.log.path

$SPLUNK_HOME\bin\scripts\splunk-winevtlog.cmd

%SPLUNK_HOME%\bin\scripts\splunk-winevtlog.cmd

@"%SPLUNK_HOME%\bin\splunk-winevtlog.exe" %* | "%SystemRoot%\System32\WindowsPowerShell\v1.0\powershell.exe" -ExecutionPolicy RemoteSigned -File "%SPLUNK_HOME%\bin\scripts\ConvertTo-JsonWinEventLog.ps1"

%SPLUNK_HOME%\bin\scripts\ConvertTo-JsonWinEventLog.ps1

Add-Type -AssemblyName "System.Web"

$xmlReaderSettings = New-Object -TypeName "System.Xml.XmlReaderSettings"
$xmlReaderSettings.ConformanceLevel = [System.Xml.ConformanceLevel]::Fragment
$xmlReaderSettings.IgnoreWhitespace = $true

$xmlStreamReader = [System.Xml.XmlReader]::Create([System.Console]::In, $xmlReaderSettings)

$xmlWriterSettings = New-Object -TypeName "System.Xml.XmlWriterSettings"
$xmlWriterSettings.ConformanceLevel = [System.Xml.ConformanceLevel]::Fragment
$xmlWriterSettings.Indent = $true
$xmlWriterSettings.IndentChars = ""

$xmlWriter = [System.Xml.XmlTextWriter]::Create([System.Console]::Out, $xmlWriterSettings)

while ($xmlStreamReader.Read()) {
    switch -Exact ($xmlStreamReader.NodeType) {
        "Element" {
            switch ($xmlStreamReader.Name) {
                "event" {
                    # expected fragment:
                    #
                    # <event stanza="WinEventLog://...">
                    # <time>...</time>
                    # <data>...</data>
                    # <source>...</source>
                    # <sourcetype>...</sourcetype>
                    # <index>...</index>
                    # </event>

                    # write the <event> element

                    $xmlWriter.WriteStartElement($xmlStreamReader.Name)

                    # write the stanza attribute

                    if ($xmlStreamReader.HasAttributes) {
                        while ($xmlStreamReader.MoveToNextAttribute()) {
                            $xmlWriter.WriteAttributeString($xmlStreamReader.Name, $xmlStreamReader.Value)
                        }

                        $result = $xmlStreamReader.MoveToElement()
                    }

                    # read and write the <time> element

                    $result = $xmlStreamReader.Read()
                    $xmlWriter.WriteStartElement($xmlStreamReader.Name)
                    $result = $xmlStreamReader.Read()
                    $xmlWriter.WriteValue($xmlStreamReader.Value)
                    $result = $xmlStreamReader.Read()
                    $xmlWriter.WriteEndElement()

                    # read and store the <data> element

                    $result = $xmlStreamReader.Read()
                    $result = $xmlStreamReader.Read()
                    $data = $xmlStreamReader.Value
                    $result = $xmlStreamReader.Read()

                    # read and store the <source> element

                    $result = $xmlStreamReader.Read()
                    $result = $xmlStreamReader.Read()
                    $source = $xmlStreamReader.Value
                    $result = $xmlStreamReader.Read()

                    # read and store the <sourcetype> element

                    $result = $xmlStreamReader.Read()
                    $result = $xmlStreamReader.Read()
                    $sourcetype = $xmlStreamReader.Value
                    $result = $xmlStreamReader.Read()

                    # modify and write the <data> element based on the <sourcetype> value
                    # modify the sourcetype value
                    
                    if ($sourcetype.startsWith("WinEventLog:")) {
                        $json = "{"
                        $stringReader = New-Object -TypeName "System.IO.StringReader" @($data)
                        # skip timestamp
                        $result = $stringReader.ReadLine()

                        while ($line = $stringReader.ReadLine()) {
                            $keyvalue = $line.Split("=", 2)
                            $key = $keyvalue[0]
                            $value = $keyvalue[1]

                            switch ($key) {
                                "Message" {
                                    $json += "`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($key) + "`":`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($value) + [System.Web.HttpUtility]::JavaScriptStringEncode($stringReader.ReadToEnd()) + "`","
                                }
                                default {
                                    $json += "`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($key) + "`":`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($value) + "`","
                                }
                            }
                        }

                        $json += "}"
                        $data = $json.Replace(",]", "]").Replace(",}", "}")

                        $sourcetype = "JsonWinEventlog"
                    }
                    elseif ($sourcetype.startsWith("XmlWinEventLog:")) {
                        $json = "{`"Event`":{"
                        $stringReader = New-Object -TypeName "System.IO.StringReader" @($data)
                        $xmlEventReader = [System.Xml.XmlReader]::Create($stringReader, $xmlReaderSettings)
                        $result = $xmlEventReader.MoveToContent()

                        while ($xmlEventReader.Read()) {
                            switch -Exact ( $xmlEventReader.NodeType) {
                                "Element" {
                                    if ($xmlEventReader.HasAttributes -or -not $xmlEventReader.IsEmptyElement) {
                                        switch -Exact ($xmlEventReader.Name) {
                                            "EventData" {
                                                $json += "`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($xmlEventReader.Name) + "`":["
                                            }
                                            "Data" {
                                                $json += "{"

                                                while ($xmlEventReader.MoveToNextAttribute()) {
                                                    $json += "`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($xmlEventReader.Name) + "`":`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($xmlEventReader.Value) + "`","
                                                }

                                                $result = $xmlEventReader.MoveToElement()
                                            }
                                            default {
                                                $json += "`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($xmlEventReader.Name) + "`":{"

                                                if ($xmlEventReader.HasAttributes) {
                                                    while ($xmlEventReader.MoveToNextAttribute()) {
                                                        $json += "`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($xmlEventReader.Name) + "`":`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($xmlEventReader.Value) + "`","
                                                    }

                                                    $result = $xmlEventReader.MoveToElement()
                                                }

                                                if ($xmlEventReader.IsEmptyElement) {
                                                    $json += "},"
                                                }
                                            }
                                        }
                                    }
                                }
                                "Text" {
                                    $json += "`"Value`":`"" + [System.Web.HttpUtility]::JavaScriptStringEncode($xmlEventReader.Value) + "`""
                                }
                                "EndElement" {
                                    if ($xmlEventReader.Name -eq "EventData") {
                                        $json += "],"
                                    }
                                    else {
                                        $json += "},"
                                    }
                                }
                            }
                        }

                        $json += "}"
                        $data = $json.Replace(",]", "]").Replace(",}", "}")

                        $sourcetype = "JsonXmlWinEventlog"
                    }

                    # write the <data> element
                    
                    $xmlWriter.WriteStartElement("data")
                    $xmlWriter.WriteValue($data)
                    $xmlWriter.WriteEndElement()

                    # write the <source> element

                    $xmlWriter.WriteStartElement("source")
                    $xmlWriter.WriteValue($source)
                    $xmlWriter.WriteEndElement()

                    # write the <sourcetype> element>

                    $xmlWriter.WriteStartElement("sourcetype")
                    $xmlWriter.WriteValue($sourcetype)
                    $xmlWriter.WriteEndElement()

                    # continue
                }
                default {
                    $xmlWriter.WriteStartElement($xmlStreamReader.Name)

                    if ($xmlStreamReader.HasAttributes) {
                        while ($xmlStreamReader.MoveToNextAttribute()) {
                            $xmlWriter.WriteAttributeString($xmlStreamReader.Name, $xmlStreamReader.Value)
                        }

                        $result = $xmlStreamReader.MoveToElement()
                    }
                }
            }
        }
        "Text" {
            $xmlWriter.WriteValue($xmlStreamReader.Value)
        }
        "EndElement" {
            $xmlWriter.WriteEndElement()
        }
    }

    $xmlWriter.Flush()
}

Sample inputs.conf with renderXml = false:

[WinEventLog://Security]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
renderXml = false

Sample inputs.conf with renderXml = true:

[WinEventLog://Security]
checkpointInterval = 5
current_only = 0
disabled = 0
start_from = oldest
renderXml = true
suppress_text = true
suppress_sourcename = true
suppress_keywords = true
suppress_type = true
suppress_task = true
suppress_opcode = true

Tying everything together generates events with either sourcetype=JsonWinEventLog or sourcetype=JsonXmlWinEventLog depending on the original sourcetype. As @PickleRick noted, your next task would be re-creating the knowledge objects provided by Splunk Add-on for Windows.

Do try this at home! But don't try it in production without sufficient testing.

If you're looking for a simpler solution, there's at least one very popular third-party product that handles transformations like this in manageable pipelines.

[PickleRick]

While all is fine and dandy (seriously, hats off; I hate powershell myself), you should _not_ touch $SPLUNK_HOME/bin. Overwriting Splunk-supplied stuff where there is no mechanism specifically provided for this (by config layering; remember that you should _not_ touch settings in default directories) is a very bad idea - it's non-maintaineable. Any UF upgrade will overwrite the changes you made. So it's not the proper way to introduce such changes. While modular inputs as such are not officially supported on UF (most probably because typically modular inputs are associated with python which is not distributed with UF installation), you might try to get away with defining your input separately in an app. But I won't guarantee it will work.

[tscroggins]

With the exception of custom scripted alert actions, which live in $SPLUNK_HOME/bin/scripts, I agree. (My memory is fuzzy here, but I don't think Splunk will run them from an app/bin directory.) This was  all a bit of evening hacking and "Can Splunk do that?" fun. Modular inputs don't have to be Python--WinEventLog isn't--but you do lose the helpfulness of the SDK, and as you say, no Splunk-supported Python interpreter is shipped with the UF.

Transforming WinEventLog with SEDCMD or INEGEST_EVAL should be simple, but transforming XmlWinEventLog requires enumerating multiple Data elements; I don't have a clever method for the latter yet. It's a shame DSP was never made widely available, and Edge Processor is still a walled garden. I generally recommend customers use a third-party product for this specific use case.

[PickleRick]

Splunk will happily run scripted input from $SPLUNK_HOME/etc/apps/<app>/bin - I run a powershell input this way.

[tscroggins]

Scripted inputs yes, scripted alert actions maybe not.

[PickleRick]

Ahh, right. The alert actions, not sripted input. But still, the docs say it's OK with placing them in an app (and that makes sense - you push alert actions, for example for ES, with deployer onto your SHC.

alert.execute.cmd = <string>
* For custom alert actions, explicitly specifies the command to run
  when the alert action is triggered. This refers to a binary or script
  in the 'bin' folder of the app that the alert action is defined in, or to a
  path pointer file, also located in the 'bin' folder.
* If a path pointer file (*.path) is specified, the contents of the file
  is read and the result is used as the command to run.
  Environment variables in the path pointer file are substituted.
* If a python (*.py) script is specified, it is prefixed with the
  bundled python interpreter.

[tscroggins]

It was circa 7.3 the last time I wrote one, but at the time, I don't think Splunk would run them from an app dir. I resorted to strace to confirm. I had a run-at-startup scripted input that would sync the alert action scripts to $SPLUNK_HOME/bin/scripts. Would I have this much fun without the workarounds? 😛

[PickleRick]

That's interesting (and I'm not being sarcastic here) because the docs as far back as 7.0 say the same -

  This refers to a binary or script
  in the bin folder of the app the alert action is defined in, or to a
  path pointer file, also located in the bin folder.

[tscroggins]

Note that the script does not handle the event stream done tag or the event unbroken attribute. I did not observe those in testing, but I did not test extensively.

[PickleRick]

I suppose, judging by the section of Answers you posted it into, you want to ingest the json-formatted windows events supplied by third party "forwarder" (whatever it is - NXLog, Kiwi, winlogbeat...).

You can ingest the events in any way you want but unless they are in one of the two formats supported by TA_windows, you're on your own with parsing and such.

See for example, the https://community.splunk.com/t5/Getting-Data-In/Connect-winlogbeat-log-format-to-Splunk-TA-Windows/m... thread for similar question.

[SplunkExplorer]

Hi @PickleRick , forgive me, I fear I explained myself bad.

Windows logs are coming directly from Domain Controllers. They are ingested using UF and they transitate through HF, so the final flow is:

DCs with UF installed -> HF -> Splunk Cloud environment

In addiction to this, the TA_windows is installed on both HF an Splunk Cloud. So, we don't want ingest data from third party forwarder; we want to know if, with this environment and the above addon installed, we are able to see logs on JSON format, when we perform searches on  SH, or we can see only Legacy and XML one because, with this environment and this addon, no other format are supported.

[PickleRick]

There is nothing to forgive. Most of us are not native English speakers so sometimes conveying your thoughts to words can be tricky 🙂

So you have your data ingested "properly" and just want to render your events on output of your search as json structures?

That's actually quite simple.

<yoursearch>
| tojson

[SplunkExplorer]

Thanks a lot, it works!

Just another last question: what about perform the change to parsing/addon?

I mean: command you shared with me works if I put it on a search when I'm logged on console. Suppose customer ask us "I want data already in JSON when I perform search, withou put command | tojson".
Is it possible to configure some parameter in TA_windows addon to achieve this? Or I can only get the 2 format XML and Legacy?

[PickleRick · Accepted solution]

No. Splunk has no concept of fields in index time (apart from indexed fields). And even if you managed to extract all files in index time (which is not achievable with xml logs since there are no xml functions working in index time) I can think of no way to wildcard fields for creating a json out of them (you can't expect all windows events to have the same field set ;-)).