topic Re: XML Parsing using SPath in Dashboards & Visualizations

XML Parsing using SPath

shan_santosh — Tue, 23 Aug 2016 15:14:36 GMT

My Windows security event looks like below

I want to get the value of element Data based on specific Name attribute. I can get this by spcifying index as below
| spath output=test path="Event.EventData.Data{2}"
| spath output=test path="Event.EventData.Data{3}"

But instead of value 2 or 3, I want to use a name as MemberSid or TargetDomainName. Can anyone help me to define path for this?

Re: XML Parsing using SPath

sundareshr — Tue, 23 Aug 2016 15:45:50 GMT

Have you tried

.. | spath output=test path="Event.EventData.Data{@TargetDomainName}"

Re: XML Parsing using SPath

somesoni2 — Tue, 23 Aug 2016 15:53:57 GMT

I don't think so the spath command supports attribute name based field reference.

Give this workaround a shot

your base search | rex mode=sed "s/(\<Data Name=\")([^\"]+)(\"\>)([^\<]+)(\<\/Data\>)/<\2>\4<\2>/g"

See this runanywhere sample

| gentimes start=-1 | eval _raw="<Event><EventData><Data Name=\"name1\">value1</Data><Data Name=\"name2\">value2</Data></EventData></Event>" | table _raw | eval temp=_raw| rex mode=sed "s/(\<Data Name=\")([^\"]+)(\"\>)([^\<]+)(\<\/Data\>)/<\2>\4<\2>/g" | spath input=_raw

Re: XML Parsing using SPath

jkat54 — Tue, 23 Aug 2016 17:38:08 GMT

Not sure if this helps or confuses:

| makeresults count=1 
| eval _raw="<Event xmlns=\"http://schemas.microsoft.com/win/2004/08/events/event\">
<EventData>
<Data Name=\"MemberName\">member</Data>
<Data Name=\"MemberSid\">ABC</Data>
<Data Name=\"TargetDomainName\">domain</Data>
<Data Name=\"MemberName2\">member</Data>
</EventData>
</Event>" 
|  spath output=name path=Event.EventData.Data{@Name} 
| mvexpand name 
| table _raw name
| appendcols
  [
| makeresults count=1 
| eval _raw="<Event xmlns=\"http://schemas.microsoft.com/win/2004/08/events/event\">
<EventData>
<Data Name=\"MemberName\">member</Data>
<Data Name=\"MemberSid\">ABC</Data>
<Data Name=\"TargetDomainName\">domain</Data>
<Data Name=\"MemberName2\">member</Data>
</EventData>
</Event>" 
| spath output=data path=Event.EventData.Data 
| mvexpand data 
| table _raw data
 ]
| search name=MemberSid OR name=TargetDomainName

Re: XML Parsing using SPath

shan_santosh — Wed, 24 Aug 2016 11:53:21 GMT

in my case I tried above with Index=myindex | rex mode=sed "s/()([^<]+)(<\/Data>)/<\2>\4<\2>/g"
This does not gave me any result.

Basically I want to extract value1 and value2 as per your example above as separate fields.

Re: XML Parsing using SPath

shan_santosh — Wed, 24 Aug 2016 11:56:14 GMT

Its not working, giving blank values

Re: XML Parsing using SPath

shan_santosh — Wed, 24 Aug 2016 12:33:20 GMT

How can I specify my index here as it says makersult has to be the first command.

Re: XML Parsing using SPath

jkat54 — Wed, 24 Aug 2016 14:45:44 GMT

Replace my "| makeresults count=1" with your main search string.

Note I used makeresults twice. Both should be replaced with your main search string.

Re: XML Parsing using SPath

jkat54 — Wed, 24 Aug 2016 14:47:53 GMT

The when you replace | makeresults on the appendcols command be sure to add the word "search" directly after the left square brackets "[" followed by your main search..

Ex.
...|appendcols [ search index=main ...

Re: XML Parsing using SPath

shan_santosh — Thu, 25 Aug 2016 07:00:19 GMT

I tried below
index=myindex
| spath output=name path=Event.EventData.Data{@Name}
| mvexpand name
| table name | appendcols
[ | search index=myindex
| spath output=data path=Event.EventData.Data
| mvexpand data
| table data ]
| search name=MemberSid OR name=TargetDomainName

but getting only name nothing for field data

Re: XML Parsing using SPath

JDukeSplunk — Thu, 25 Aug 2016 14:25:01 GMT

I hate to ask the question, but why wouldn't you just use the Splunk TA Windows app to capture Windows EventLogs? It has built-in transforms and props.conf files to parse and extract WinEventLogs as well as perfmon and other nice things.

https://splunkbase.splunk.com/app/742/

This is our modfied inputs.conf with some customized perfmon and security log filtering. If you do end up using this feel free to reach out, I can send you some searches.

[default]
evt_dc_name =
evt_dns_name =


###### OS Logs ######
[WinEventLog://Application]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
index = wineventlog
renderXml=false

[WinEventLog://Security]
disabled = 0
start_from = oldest
current_only = 0
evt_resolve_ad_obj = 1
checkpointInterval = 5
blacklist1 = EventCode="4662" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist2 = EventCode="566" Message="Object Type:\s+(?!groupPolicyContainer)"
blacklist3 = EventCode="5156" Message="(s|S)plunkd.exe"
blacklist4 = EventCode="4656" Message="(d|D)esktop.ini"
blacklist5 = EventCode="4656" Message="PlugPlaySecurityObject"
index = wineventlog
renderXml=false

[WinEventLog://System]
disabled = 0
start_from = oldest
current_only = 0
checkpointInterval = 5
index = wineventlog
renderXml=false


###### Windows Update Log ######
[monitor://$WINDIR\WindowsUpdate.log]
disabled = 0
sourcetype = WindowsUpdateLog
index = windows

###### Scripted Input (See also wmi.conf)
[script://.\bin\win_listening_ports.bat]
disabled = 1
## Run once per hour
interval = 3600
sourcetype = Script:ListeningPorts
index = windows

[script://.\bin\win_installed_apps.bat]
disabled = 0
## Run once per day
interval = 86400
sourcetype = Script:InstalledApps
index = windows

[perfmon://CPU Load]
index = perfmon
counters = % Processor Time;% User Time
instances = _Total
interval = 60
object = Processor

[perfmon://Available Memory]
index = perfmon
counters = Available Bytes
interval = 60
object = Memory

[perfmon://Free Disk Space]
index = perfmon
counters = Free Megabytes;% Free Space
instances = C:;D:;E:;F:
# 15 minutes
interval = 900
object = LogicalDisk

[perfmon://Network Interface]
index = perfmon
counters = Bytes Received/sec;Bytes Sent/sec
instances = *
interval = 60
object = Network Interface

[perfmon://LocalPhysicalDisk]
index = perfmon
# 5 minutes
interval = 300
object = PhysicalDisk
counters = Disk Bytes/sec; % Disk Read Time; % Disk Write Time; % Disk Time ;Avg. Disk Bytes/Read; Avg. Disk Bytes/Transfer; Avg. Disk Bytes/Write; Avg. Disk Queue Length; Avg. Disk Read Queue Length; Avg. Disk Write Queue Length; Avg. Disk sec/Read; Avg. Disk sec/Transfer; Avg. Disk sec/Write
instances = *
disabled = 0


[admon://default]
disabled = 1
monitorSubtree = 1

[WinRegMon://default]
disabled = 1
interval = 360
hive = .*
proc = .*
type = rename|set|delete|create
index = perfmon

[WinRegMon://hkcu_run]
disabled = 1
interval = 360
hive = \\REGISTRY\\USER\\.*\\Software\\Microsoft\\Windows\\CurrentVersion\\Run\\.*
proc = .*
type = set|create|delete|rename
index = perfmon

[WinRegMon://hklm_run]
disabled = 1
interval = 360
hive = \\REGISTRY\\MACHINE\\SOFTWARE\\Microsoft\\Windows\\CurrentVersion\\Run\\.*
proc = .*
type = set|create|delete|rename
index = perfmon

Re: XML Parsing using SPath

shan_santosh — Tue, 29 Sep 2020 10:45:42 GMT

Thanks for your comments on this and showing readiness to help by proving search string etc.

However, I am already using Splunk TA Windows application to get windows security events data using below stanza in inputs.conf file

Monitors Windows Security Events

[WinEventLog://Security]
current_only = 1
renderXml = 1
suppress_text = 0
checkpointInterval = 30
evt_resolve_ad_obj = 1
evt_dc_name = ap.com, na.com, eu.com,
whitelist = 4732,4733
index = myIndex
disabled = 0

But the problem is on one of my Splunk server 6.2 version, when I search index=myIndex it automatically extracts all the fields including XML attribute names etc. Where as on another Splunk server version 6.4.3 it does not extracts all fields automatically.

I have also set KV_Mode = XML on my Splunk Indexer but still its not working. May be some thing is missing so Splunk 6.4.3 is not automatically extracting XML fields while search or during indexing.

Re: XML Parsing using SPath

jkat54 — Fri, 26 Aug 2016 19:29:54 GMT

hmm it worked with your data on my splunk...

Not sure if it matters but you had an extra pipe in the appendcols. See if this works:

index=myindex
| spath output=name path=Event.EventData.Data{@Name}
| mvexpand name
| table name | appendcols
[ search index=myindex
| spath output=data path=Event.EventData.Data
| mvexpand data
| table data ]
| search name=MemberSid OR name=TargetDomainName

Re: XML Parsing using SPath

acharlieh — Sun, 28 Aug 2016 17:57:54 GMT

I have also set KV_Mode = XML on my Splunk Indexer but still its not working.

Two problems with this statement... First, attribute names in .conf files are case sensitive, attribute values sometimes can be case sensitive. Therefore this props.conf setting should be KV_MODE = xml not KV_Mode = XML per the props.conf spec. Secondly, KV_MODE is a search time attribute, therefore it should to be set for your sourcetype on your search head, ideally in the app where you're performing your search if not globally.

Re: XML Parsing using SPath

acharlieh — Sun, 28 Aug 2016 18:25:36 GMT

Does it have to be spath? If you're open to using xpath instead you could do something like this:

| xpath outfield=MemberSid "//*[local-name()='Data' and namespace-uri()='http://schemas.microsoft.com/win/2004/08/events/event' and @Name='MemberSid']"

It is indeed a bit more complex of a query, thanks to unprefixed XML namespaces, and the corresponding xpath behaviour as pointed out in this stack overflow answer, but you can get the value of any data element with a specific Name attribute this way.

Dummy Test:

 | makeresults | eval _raw="<Event xmlns=\"http://schemas.microsoft.com/win/2004/08/events/event\">
 <EventData>
 <Data Name=\"MemberName\">member</Data>
 <Data Name=\"MemberSid\">ABC</Data>
 <Data Name=\"TargetDomainName\">domain</Data>
 <Data Name=\"MemberName2\">member</Data>
 </EventData>
 </Event>" | xpath outfield=MemberSid "//*[local-name()='Data' and namespace-uri()='http://schemas.microsoft.com/win/2004/08/events/event' and @Name='MemberSid']"