Hey all,
I am really struggling to create a parser for a specific section of the Windows-TerminalServices-Gateway/Operational Event log.
Everything I have added to props & transforms appears to be correct, and I can get the sections I want to parse via SPL but whenever I add it to the sourcetype and refresh nothing changes.
inputs.conf on the Windows 2016 Server
[WinEventLog://Microsoft-Windows-TerminalServices-Gateway/Operational]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml = true
index = windows
[WinEventLog://Microsoft-Windows-TerminalServices-Gateway/Admin]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
renderXml = true
index = windows
props.conf on search head
[XmlWinEventLog:Microsoft-Windows-TerminalServices-Gateway/Operational]
rename = XmlWinEventLog
[XmlWinEventLog:Microsoft-Windows-TerminalServices-Gateway/Admin]
rename = XmlWinEventLog
[xmlwineventlog]
REPORT-0xml_kv_extract = system_props_xml_kv,system_props_xml_attributes,eventdata_xml_data,rendering_info_xml_data,userdata_xml_kv,userdata_xml_attributes
[XmlWinEventLog]
REPORT-0xml_kv_extract = system_props_xml_kv,system_props_xml_attributes,eventdata_xml_data,rendering_info_xml_data,userdata_xml_kv,userdata_xml_attributes
transforms.conf on search head
[userdata_xml_kv]
# Extracts anything in the form of <tag>value</tag> as tag::value
SOURCE_KEY = UserData_Xml
REGEX = (?ms)<(\w*)>([^<]*)<\/\1>
FORMAT = $1::$2
MV_ADD = 1
[userdata_xml_attributes]
# Extracts values from following fields:
# EventInfo: xmlns
SOURCE_KEY = UserData_Xml
REGEX = (?ms)([^\s=]+)\s*=\s*(\'[^<\']*\'|"[^<"]*")
FORMAT = $1::$2
MV_ADD = 1
Assuming you have the raw XML above ingested, you should be able to parse it using SPL like below at search time:
| makeresults
| eval _raw = "<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'>
<System>
<Provider Name='Microsoft-Windows-TerminalServices-Gateway' Guid='{4D5AE6A1-C7C8-4E6D-B840-4D8080B42E1B}'/>
<EventID>303</EventID>
<Version>0</Version>
<Level>4</Level>
<Task>3</Task>
<Opcode>44</Opcode>
<Keywords>0x4000000001000000</Keywords>
<TimeCreated SystemTime='2020-04-16T16:16:53.376065200Z'/>
<EventRecordID>17096</EventRecordID>
<Correlation ActivityID='{320C9F53-4311-4E6E-8CA7-243377CB0000}'/>
<Execution ProcessID='4676' ThreadID='13572'/>
<Channel>Microsoft-Windows-TerminalServices-Gateway/Operational</Channel>
<Computer>MACHINE.example.com</Computer>
<Security UserID='S-1-5-20'/>
</System>
<UserData>
<EventInfo xmlns='aag'>
<Username>EXAMPLE\testuser</Username>
<IpAddress>10.0.0.5</IpAddress>
<AuthType></AuthType>
<Resource>machine2</Resource>
<BytesReceived>134859</BytesReceived>
<BytesTransfered>116524</BytesTransfered>
<SessionDuration>39</SessionDuration>
<ConnectionProtocol>HTTP</ConnectionProtocol>
<ErrorCode>1226</ErrorCode>
</EventInfo>
</UserData>
</Event>"
| spath
| rename Event.UserData.EventInfo.Username as UserName
| rename Event.UserData.EventInfo.IpAddress as IpAddress
| rename Event.UserData.EventInfo.Resource as Resource
| rename Event.UserData.EventInfo.BytesReceived as BytesReceived
| rename Event.UserData.EventInfo.BytesTransfered as BytesTransfered
| rename Event.UserData.EventInfo.SessionDuration as SessionDuration
| rename Event.UserData.EventInfo.ConnectionProtocol as ConnectionProtocol
| rename Event.UserData.EventInfo.ErrorCode as ErrorCode
| table UserName, IpAddress, Resource,BytesReceived,BytesTransfered,SessionDuration,ConnectionProtocol,ErrorCode
I don't know the details of your setup on the forwarder so couldn't comment on that.
Thanks for the response. I do not have issues parsing with SPL from search. the issue comes in when I tried to add the parsing logic to the Splunk_TA_windows add-on as a multi-value transform.
I am still stuck, any recommendations to try would be greatly appreciated.
Hi there, questions that might help frame the issue a bit better:
This is a standalone splunk server with UF on each endpoint, no HF.
I have successfully collected the logs I want with a UF but cant get the fields parsed that I am trying to get out.
I am adding the configurations above to the Splunk_TA_windows/local directory
Here is an example of what one log looks like. specifically from that log I am trying to extract the fields between the UserData markup.
<Event xmlns='http://schemas.microsoft.com/win/2004/08/events/event'><System><Provider Name='Microsoft-Windows-TerminalServices-Gateway' Guid='{4D5AE6A1-C7C8-4E6D-B840-4D8080B42E1B}'/><EventID>303</EventID><Version>0</Version><Level>4</Level><Task>3</Task><Opcode>44</Opcode><Keywords>0x4000000001000000</Keywords><TimeCreated SystemTime='2020-04-16T16:16:53.376065200Z'/><EventRecordID>17096</EventRecordID><Correlation ActivityID='{320C9F53-4311-4E6E-8CA7-243377CB0000}'/><Execution ProcessID='4676' ThreadID='13572'/><Channel>Microsoft-Windows-TerminalServices-Gateway/Operational</Channel><Computer>MACHINE.example.com</Computer><Security UserID='S-1-5-20'/></System><UserData><EventInfo xmlns='aag'><Username>EXAMPLE\testuser</Username><IpAddress>10.0.0.5</IpAddress><AuthType></AuthType><Resource>machine2</Resource><BytesReceived>134859</BytesReceived><BytesTransfered>116524</BytesTransfered><SessionDuration>39</SessionDuration><ConnectionProtocol>HTTP</ConnectionProtocol><ErrorCode>1226</ErrorCode></EventInfo></UserData></Event>