Is there a better way to parse DTS Compliant IAS logs (xml format)?

[pjaguilarjr]

DTS Compliant IAS logs are IAS logs wrapped in XML. Additional info: http://technet.microsoft.com/en-us/library/cc771748(v=ws.10).asp

I can extract field names at search time with |xmlkv but I that's not permanent. I've tried putting

KV_MODE=xml

into props.conf a few different ways and confirmed with btool, but apparently this format isn't true xml so splunk is ignoring it. Any suggestions? Here's a sample event:

<Timestamp data_type="4">12/22/2009 15:06:56.609</Timestamp> 
                <Computer-Name data_type="1">NAP-IAS2</Computer-Name>
                <Event-Source data_type="1">IAS</Event-Source>
                <Acct-Session-Id data_type="2">B3BA359F48CEDE4E9F78E5B3158F3B877E744D735B83CA01</Acct-Session-Id>
                <Class data_type="1">311 1 2001:4898:b0:3007:492e:957a:d44d:7093 12/16/2009 04:32:04 145361</Class>
                <MS-Extended-Quarantine-State data_type="0">0</MS-Extended-Quarantine-State>
                <MS-Quarantine-State data_type="0">0</MS-Quarantine-State>
                <Client-IPv6-Address data_type="5">2001:4898:b0:3007:6cc0:9514:d2ff:cdcf</Client-IPv6-Address>
                <Client-Vendor data_type="0">0</Client-Vendor>
                <Client-Friendly-Name data_type="1">NAP-HRA2</Client-Friendly-Name>
                <Proxy-Policy-Name data_type="1">HRA</Proxy-Policy-Name>
                <Provider-Type data_type="0">1</Provider-Type>
                <Quarantine-Session-Id data_type="1">{9F35BAB3-CE48-4EDE-9F78-E5B3158F3B87} - 2009-12-22 23:06:53.319Z</Quarantine-Session-Id>
                <Machine-Inventory data_type="1">6.1.7600 0.0 x86 Workstation</Machine-Inventory>
                <Fully-Qualified-Machine-Name data_type="1">CONTOSO\CLIENT1</Fully-Qualified-Machine-Name>
                <Authentication-Type data_type="0">7</Authentication-Type>
                <System-Health-Result data_type="1">Windows Security Health Validator:Compliant:No Data:None[]:(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - ):(0x0 - )</System-Health-Result>
                <System-Health-ResultEx data_type="1">
                                <SHV-Name data_type="1">Windows Security Health Validator</SHV-Name>
                                <Config-ID data_type="0">0</Config-ID>
                                <Config-Friendly-Name data_type="1"></Config-Friendly-Name>
                                <Health-Result data_type="1">Compliant</Health-Result>
                                <Extended-Isolation-State data_type="1">No Data</Extended-Isolation-State>
                                <Failure-Category data_type="1">None</Failure-Category>
                                <Failure-Category-String data_type="1"></Failure-Category-String>
                                <Compliance-Results data_type="1"></Compliance-Results>
                </System-Health-ResultEx>
                <NP-Policy-Name data_type="1">ias2-HRA-NAPSTIR-Red-Compliant</NP-Policy-Name>
                <Quarantine-Update-Non-Compliant data_type="0">0</Quarantine-Update-Non-Compliant>
                <Framed-Protocol data_type="0">1</Framed-Protocol>
                <Service-Type data_type="0">2</Service-Type>
                <Packet-Type data_type="0">2</Packet-Type>
                <Reason-Code data_type="0">0</Reason-Code>
</Event>

[hburton]

I'm seeing the same thing. 

props.conf looks like.

[DTS_Compliant_IAS]

BREAK_ONLY_BEFORE = <Event>

NO_BINARY_CHECK = 1

TIME_FORMAT = %m/%d/%Y %H:%M:%S.%3N

TIME_PREFIX = <Timestamp data_type="4">

pulldown_type = 1

KV_MODE=xml

inputs.conf looks like

[monitor://C:\Windows\System32\LogFiles\IN*.log]

sourcetype = DTS_Compliant_IAS

index = radius

disabled = 0

If I search with |xmlkv my field are extracted, same as original poster. Otherwise, it's just the full xml string.

[isoutamo]

Hi

you should use KV_MODE on search time extractions and INDEXED_EXTRACTIONS on index time extractions.

https://docs.splunk.com/Documentation/Splunk/7.3.3/Admin/Propsconf

r. Ismo

[hburton]

Thank you soutamo.

Edited Reply:

Now that I know the difference (thanks again. I'm brand new here) I see that it doesn't I considered writing a small python file monitor to convert the xml to json, and I may do that, but it looks like the Splunk proper way to do it is to use the KV_MODE=xml on a config file at the search head. We are using Cloud as our search head, so I don't know how to access my props.conf. Is there a way to do the same thing in settings somehow?

[isoutamo]

Hi

especially in cloud environment you must have your own test environment for do data on boarding tests. Install free version and use trial license or request developer license from Splunk. After you have do your tests successfully you should create apps for those props.conf etc and do service request to Splunk to install that app to your Splunk cloud environment. There are instructions how you show create that app etc. 

But as I said, you must have your own test instance where you develop these before you could add those to cloud.

r. Ismo

[microwavelunch1]

Did you ever resolve this? I have the same issue, despite putting KV_MODE=xml on both UF and indexer, the events do not get parsed as XML, but if you do a search time parse it works perfectly...

[SplunkerPaul]

You need to add KV_MODE=xml  in props.conf on your search head. This is a search time field extraction configuration.

[norbert_hamel]

Hi,

I have checked this setting in props.conf

[DTS_Compliant_IAS]
BREAK_ONLY_BEFORE = <Event>
NO_BINARY_CHECK = 1
TIME_FORMAT = %m/%d/%Y %H:%M:%S.%3N
TIME_PREFIX = <Timestamp data_type="4">
pulldown_type = 1
KV_MODE=xml

Additionally, your example is missing the opening tag, so I have added this to the test log.

Using this settings the search for sourcetype=DTS_Compliant_IAS extracts all fields for me.