Splunk Search
Highlighted

How can I correctly parse time from the XML field?

Explorer

I am attempting to extract Time using TIMEFORMAT and TIMEPREFIX in props.conf. Would like to understand how to correctly parse the Time from the GMTTime XML field. The original message is read from a file and sent using a universal forwarder. The inputs.conf on the universal forwarder looks like this:

[monitor:///opt/xml/events.txt]
disabled = false
sourcetype = epo
host = lab-epo

The original message looks like this:

<29>1 2017-08-18T02:50:19.0Z LAB-EPO EPOEvents - EventFwd [agentInfo@3401 tenantId="1"] <?xml version="1.0" encoding="UTF-8"?><EPOevent><MachineInfo><MachineName>LAB-WIN7-02</MachineName><AgentGUID>{b37ff914-XXXX-XXXX-8740-91aa851f0e3d}</AgentGUID><IPAddress>192.XXX.XXX.XXX</IPAddress><OSName>Windows 7</OSName><UserName>SYSTEM</UserName><TimeZoneBias>240</TimeZoneBias><RawMACAddress>XXXXXXXX</RawMACAddress></MachineInfo><SoftwareInfo ProductName="McAfee Endpoint Security" ProductVersion="10.5.0" ProductFamily="TVD"><CommonFields><Analyzer>ENDP_AM_1050</Analyzer><AnalyzerName>McAfee Endpoint Security</AnalyzerName><AnalyzerVersion>10.5.0</AnalyzerVersion><AnalyzerHostName>LAB-WIN7-02</AnalyzerHostName><AnalyzerEngineVersion>XXXX.7806</AnalyzerEngineVersion><AnalyzerDetectionMethod>On-Access Scan</AnalyzerDetectionMethod><AnalyzerDATVersion>3075.0</AnalyzerDATVersion></CommonFields><Event><EventID>1278</EventID><Severity>3</Severity><GMTTime>2017-08-18T14:48:53</GMTTime><CommonFields><ThreatCategory>av.detect</ThreatCategory><ThreatEventID>1278</ThreatEventID><ThreatSeverity>2</ThreatSeverity><ThreatName>EICAR test file</ThreatName><ThreatType>test</ThreatType><DetectedUTC>2017-08-18T14:48:53Z</DetectedUTC><ThreatActionTaken>IDS_ALERT_ACT_TAK_DEL</ThreatActionTaken><ThreatHandled>True</ThreatHandled><SourceHostName>LAB-WIN7-02</SourceHostName><SourceProcessName>C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE</SourceProcessName><TargetHostName>LAB-WIN7-02</TargetHostName><TargetUserName>LAB-WIN7-02\xadmin</TargetUserName><TargetFileName>C:\USERS\XADMIN\DOWNLOADS\Unconfirmed 408214.crdownload</TargetFileName></CommonFields><CustomFields target="EPExtendedEventMT"><BladeName>IDS_BLADE_NAME_SPB</BladeName><AnalyzerContentCreationDate>2017-08-16T13:00:00Z</AnalyzerContentCreationDate><AnalyzerGTIQuery>False</AnalyzerGTIQuery><ThreatDetectedOnCreation>False</ThreatDetectedOnCreation><TargetName>Unconfirmed 408214.crdownload</TargetName><TargetPath>C:\USERS\XADMIN\DOWNLOADS</TargetPath><TargetHash>44d88612fea8a8f36de82e1278abb02f</TargetHash><TargetFileSize>68</TargetFileSize><TargetModifyTime>2017-08-18T14:48:53Z</TargetModifyTime><TargetAccessTime>2017-08-18T14:48:53Z</TargetAccessTime><TargetCreateTime>2017-08-18T14:48:53Z</TargetCreateTime><Cleanable>False</Cleanable><TaskName>IDS_OAS_TASK_NAME</TaskName><FirstAttemptedAction>IDS_ALERT_THACT_ATT_CLE</FirstAttemptedAction><FirstActionStatus>False</FirstActionStatus><SecondAttemptedAction>IDS_ALERT_THACT_ATT_DEL</SecondAttemptedAction><SecondActionStatus>True</SecondActionStatus><AttackVectorType>4</AttackVectorType><DurationBeforeDetection>0</DurationBeforeDetection><NaturalLangDescription>IDS_NATURAL_LANG_OAS_DETECTION_DEL|TargetName=Unconfirmed 408214.crdownload|TargetPath=C:\USERS\XADMIN\DOWNLOADS|ThreatName=EICAR test file|SourceProcessName=C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE|ThreatType=test|TargetUserName=LAB-WIN7-02\xadmin</NaturalLangDescription><AccessRequested></AccessRequested><DetectionMessage>IDS_OAS_DEFAULT_THREAT_MESSAGE</DetectionMessage><AMCoreContentVersion>3075.0</AMCoreContentVersion></CustomFields></Event></SoftwareInfo></EPOevent>

The props.conf on the receiving indexer looks like this:

[epo]
SEDCMD-replace = s/\<29\>[^\>]*\>\n*//g
KV_MODE = xml
TIME_PREFIX = \<EPOevent.SoftwareInfo.Event.GMTTime\>
TIME_FORMAT = %Y-%m-%dT%H:%M:%S

I have also tried:

[epo]
SEDCMD-replace = s/\<29\>[^\>]*\>\n*//g
KV_MODE = xml
TIME_PREFIX = \<GMTTime\>
TIME_FORMAT = %Y-%m-%dT%H:%M:%S

Via search, the event looks like the following after SEC_CMD as parse the message:

<EPOevent><MachineInfo><MachineName>LAB-WIN7-02</MachineName><AgentGUID>{b37ff914-XXXX-xxxx-XXXX-91aa851fXXXX}</AgentGUID><IPAddress>192.xXX.xxx.102</IPAddress><OSName>Windows 7</OSName><UserName>SYSTEM</UserName><TimeZoneBias>240</TimeZoneBias><RawMACAddress>000c29xxxxxx</RawMACAddress></MachineInfo><SoftwareInfo ProductName="McAfee Endpoint Security" ProductVersion="10.5.0" ProductFamily="TVD"><CommonFields><Analyzer>ENDP_AM_1050</Analyzer><AnalyzerName>McAfee Endpoint Security</AnalyzerName><AnalyzerVersion>10.5.0</AnalyzerVersion><AnalyzerHostName>LAB-WIN7-02</AnalyzerHostName><AnalyzerEngineVersion>5900.7806</AnalyzerEngineVersion><AnalyzerDetectionMethod>On-Access Scan</AnalyzerDetectionMethod><AnalyzerDATVersion>3075.0</AnalyzerDATVersion></CommonFields><Event><EventID>1278</EventID><Severity>3</Severity><GMTTime>2017-08-18T14:48:53</GMTTime><CommonFields><ThreatCategory>av.detect</ThreatCategory><ThreatEventID>1278</ThreatEventID><ThreatSeverity>2</ThreatSeverity><ThreatName>EICAR test file</ThreatName><ThreatType>test</ThreatType><DetectedUTC>2017-08-18T14:48:53Z</DetectedUTC><ThreatActionTaken>IDS_ALERT_ACT_TAK_DEL</ThreatActionTaken><ThreatHandled>True</ThreatHandled><SourceHostName>LAB-WIN7-02</SourceHostName><SourceProcessName>C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE</SourceProcessName><TargetHostName>LAB-WIN7-02</TargetHostName><TargetUserName>LAB-WIN7-02\xadmin</TargetUserName><TargetFileName>C:\USERS\XADMIN\DOWNLOADS\Unconfirmed 408214.crdownload</TargetFileName></CommonFields><CustomFields target="EPExtendedEventMT"><BladeName>IDS_BLADE_NAME_SPB</BladeName><AnalyzerContentCreationDate>2017-08-16T13:00:00Z</AnalyzerContentCreationDate><AnalyzerGTIQuery>False</AnalyzerGTIQuery><ThreatDetectedOnCreation>False</ThreatDetectedOnCreation><TargetName>Unconfirmed 408214.crdownload</TargetName><TargetPath>C:\USERS\XADMIN\DOWNLOADS</TargetPath><TargetHash>44d88612fea8a8f36de82e1278abb02f</TargetHash><TargetFileSize>68</TargetFileSize><TargetModifyTime>2017-08-18T14:48:53Z</TargetModifyTime><TargetAccessTime>2017-08-18T14:48:53Z</TargetAccessTime><TargetCreateTime>2017-08-18T14:48:53Z</TargetCreateTime><Cleanable>False</Cleanable><TaskName>IDS_OAS_TASK_NAME</TaskName><FirstAttemptedAction>IDS_ALERT_THACT_ATT_CLE</FirstAttemptedAction><FirstActionStatus>False</FirstActionStatus><SecondAttemptedAction>IDS_ALERT_THACT_ATT_DEL</SecondAttemptedAction><SecondActionStatus>True</SecondActionStatus><AttackVectorType>4</AttackVectorType><DurationBeforeDetection>0</DurationBeforeDetection><NaturalLangDescription>IDS_NATURAL_LANG_OAS_DETECTION_DEL|TargetName=Unconfirmed 408214.crdownload|TargetPath=C:\USERS\XADMIN\DOWNLOADS|ThreatName=EICAR test file|SourceProcessName=C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE|ThreatType=test|TargetUserName=LAB-WIN7-02\xadmin</NaturalLangDescription><AccessRequested></AccessRequested><DetectionMessage>IDS_OAS_DEFAULT_THREAT_MESSAGE</DetectionMessage>
<AMCoreContentVersion>3075.0</AMCoreContentVersion></CustomFields></Event></SoftwareInfo></EPOevent>

The indexer is still applying a timestamp of when it receives the message verse using GMTTime. Here is a formatted view of what splunk sees, e.g. the nested XML:

EPOevent.MachineInfo.AgentGUID
    {b37ff914-83f4-4b48-8740-XXXXXXXXXX}    
EPOevent.MachineInfo.IPAddress
    192.xxx.xxx.XXX 
EPOevent.MachineInfo.MachineName
    LAB-WIN7-02 
EPOevent.MachineInfo.OSName
    Windows 7   
EPOevent.MachineInfo.RawMACAddress
    000c29fXXXXX
EPOevent.MachineInfo.TimeZoneBias
    240 
EPOevent.MachineInfo.UserName
    SYSTEM  
EPOevent.SoftwareInfo.CommonFields.Analyzer
    ENDP_AM_1050    
EPOevent.SoftwareInfo.CommonFields.AnalyzerDATVersion
    3075.0  
EPOevent.SoftwareInfo.CommonFields.AnalyzerDetectionMethod
    On-Access Scan  
EPOevent.SoftwareInfo.CommonFields.AnalyzerEngineVersion
    5900.7806   
EPOevent.SoftwareInfo.CommonFields.AnalyzerHostName
    LAB-WIN7-02 
EPOevent.SoftwareInfo.CommonFields.AnalyzerName
    McAfee Endpoint Security    
EPOevent.SoftwareInfo.CommonFields.AnalyzerVersion
    10.5.0  
EPOevent.SoftwareInfo.Event.CommonFields.DetectedUTC
    2017-08-18T14:48:53Z    
EPOevent.SoftwareInfo.Event.CommonFields.SourceHostName
    LAB-WIN7-02 
EPOevent.SoftwareInfo.Event.CommonFields.SourceProcessName
    C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE 
EPOevent.SoftwareInfo.Event.CommonFields.TargetFileName
    C:\USERS\XADMIN\DOWNLOADS\Unconfirmed 408214.crdownload 
EPOevent.SoftwareInfo.Event.CommonFields.TargetHostName
    LAB-WIN7-02 
EPOevent.SoftwareInfo.Event.CommonFields.TargetUserName
    LAB-WIN7-02\xadmin  
EPOevent.SoftwareInfo.Event.CommonFields.ThreatActionTaken
    IDS_ALERT_ACT_TAK_DEL   
EPOevent.SoftwareInfo.Event.CommonFields.ThreatCategory
    av.detect   
EPOevent.SoftwareInfo.Event.CommonFields.ThreatEventID
    1278    
EPOevent.SoftwareInfo.Event.CommonFields.ThreatHandled
    True    
EPOevent.SoftwareInfo.Event.CommonFields.ThreatName
    EICAR test file 
EPOevent.SoftwareInfo.Event.CommonFields.ThreatSeverity
    2   
EPOevent.SoftwareInfo.Event.CommonFields.ThreatType
    test    
EPOevent.SoftwareInfo.Event.CustomFields.AMCoreContentVersion
    3075.0  
EPOevent.SoftwareInfo.Event.CustomFields.AnalyzerContentCreationDate
    2017-08-16T13:00:00Z    
EPOevent.SoftwareInfo.Event.CustomFields.AnalyzerGTIQuery
    False   
EPOevent.SoftwareInfo.Event.CustomFields.AttackVectorType
    4   
EPOevent.SoftwareInfo.Event.CustomFields.BladeName
    IDS_BLADE_NAME_SPB  
EPOevent.SoftwareInfo.Event.CustomFields.Cleanable
    False   
EPOevent.SoftwareInfo.Event.CustomFields.DetectionMessage
    IDS_OAS_DEFAULT_THREAT_MESSAGE  
EPOevent.SoftwareInfo.Event.CustomFields.DurationBeforeDetection
    0   
EPOevent.SoftwareInfo.Event.CustomFields.FirstActionStatus
    False   
EPOevent.SoftwareInfo.Event.CustomFields.FirstAttemptedAction
    IDS_ALERT_THACT_ATT_CLE 
EPOevent.SoftwareInfo.Event.CustomFields.NaturalLangDescription
    IDS_NATURAL_LANG_OAS_DETECTION_DEL|TargetName=Unconfirmed 408214.crdownload|TargetPath=C:\USERS\XADMIN\DOWNLOADS|ThreatName=EICAR test file|SourceProcessName=C:\PROGRAM FILES (X86)\GOOGLE\CHROME\APPLICATION\CHROME.EXE|ThreatType=test|TargetUserName=LAB-WIN7-02\xadmin 
EPOevent.SoftwareInfo.Event.CustomFields.SecondActionStatus
    True    
EPOevent.SoftwareInfo.Event.CustomFields.SecondAttemptedAction
    IDS_ALERT_THACT_ATT_DEL 
EPOevent.SoftwareInfo.Event.CustomFields.TargetAccessTime
    2017-08-18T14:48:53Z    
EPOevent.SoftwareInfo.Event.CustomFields.TargetCreateTime
    2017-08-18T14:48:53Z    
EPOevent.SoftwareInfo.Event.CustomFields.TargetFileSize
    68  
EPOevent.SoftwareInfo.Event.CustomFields.TargetHash
    44d88612fea8a8f36de82e1278abb02f    
EPOevent.SoftwareInfo.Event.CustomFields.TargetModifyTime
    2017-08-18T14:48:53Z    
EPOevent.SoftwareInfo.Event.CustomFields.TargetName
    Unconfirmed 408214.crdownload   
EPOevent.SoftwareInfo.Event.CustomFields.TargetPath
    C:\USERS\XADMIN\DOWNLOADS   
EPOevent.SoftwareInfo.Event.CustomFields.TaskName
    IDS_OAS_TASK_NAME   
EPOevent.SoftwareInfo.Event.CustomFields.ThreatDetectedOnCreation
    False   
EPOevent.SoftwareInfo.Event.CustomFields{@target}
    EPExtendedEventMT   
EPOevent.SoftwareInfo.Event.EventID
    1278    
EPOevent.SoftwareInfo.Event.GMTTime
    2017-08-18T14:48:53 
EPOevent.SoftwareInfo.Event.Severity
    3   
EPOevent.SoftwareInfo{@ProductFamily}
    TVD 
EPOevent.SoftwareInfo{@ProductName}
    McAfee Endpoint Security    
EPOevent.SoftwareInfo{@ProductVersion}
    10.5.0  
timestamp
    none    
Time            
_time   
    2017-08-18T10:50:32.000-04:00   
Default 
host
    lab-epo 
index
    main    
linecount
    1   
punct
    <><><>--</><>{----}</><>...</><>_</><></><></><></  
source
    /opt/xml/events.txt     
sourcetype
    epo
splunk_server
    lab-splunk-01
0 Karma
Highlighted

Re: How can I correctly parse time from the XML field?

SplunkTrust
SplunkTrust

Try without escape chars.

[epo]
SEDCMD-replace = s/\<29\>[^\>]*\>\n*//g
KV_MODE = xml
TIME_PREFIX = <GMTTime>
TIME_FORMAT = %Y-%m-%dT%H:%M:%S
---
If this reply helps you, an upvote would be appreciated.

View solution in original post