How to extract same set of values from different log format?

ravir_jbp — Wed, 14 Dec 2022 16:19:22 GMT

EventAgentLogin

==================

2022-12-14 06:39:03.875 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory : Endpoint 'SERVER1' received message ''EventAgentLogin' (73) attributes:

AttributeReasons [bstr] = KVList:

'referrer' [str] = "https://aaa.test.com"

'clientApp' [str] = "asdfsfdf"

'location' [str] = "server"

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0077MP"

'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744434548

TimeStamp:

AttributeTimeinSecs [int] = 1671021543

AttributeTimeinuSecs [int] = 882837'

-------------------------------------------------------------------------------------

agent ready state"

=======================

2022-12-14 07:59:12.764 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory : Endpoint 'SERVER1' received message ''EventAgentReady' (75) attributes:

AttributeReasons [bstr] = KVList:

'site' [str] = "CTC"

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744780812

TimeStamp:

AttributeTimeinSecs [int] = 1671026352

AttributeTimeinuSecs [int] = 766178'

--------------------------------------------------------

EventAgentNotReady

==================

2022-12-14 08:01:31.602 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory : Endpoint 'SERVER1' received message ''EventAgentNotReady' (76) attributes:

AttributeReasons [bstr] = KVList:

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744808316

TimeStamp:

----------------------------------------------------------

Training

==========

2022-12-14 08:02:47.211 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory : Endpoint 'SERVER1' received message ''EventAgentNotReady' (76) attributes:

AttributeReasons [bstr] = KVList:

'ReasonCode' [str] = "Training"

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744821504

TimeStamp:

AttributeTimeinSecs [int] = 1671026567

AttributeTimeinuSecs [int] = 209306'

-------------------------------------------------------------------------

"EventAgentNotReady" AND "Break"

====================================

2022-12-14 08:04:34.025 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory : Endpoint 'SERVER1' received message ''EventAgentNotReady' (76) attributes:

AttributeReasons [bstr] = KVList:

'ReasonCode' [str] = "Break"

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744838251

TimeStamp:

AttributeTimeinSecs [int] = 1671026674

AttributeTimeinuSecs [int] = 24284'

----------------------------------------------------------------------------------

AfterCallWork

===============

2022-12-14 08:07:31.310 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory : Endpoint 'SERVER1' received message ''EventAgentNotReady' (76) attributes:

AttributeReasons [bstr] = KVList:

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

'ReasonCode' [str] = "ManualSetACWPeriod"

'WrapUpTime' [str] = "untimed"

'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 3 [AfterCallWork]

AttributeEventSequenceNumber [long] = 744864731

TimeStamp:

AttributeTimeinSecs [int] = 1671026851

AttributeTimeinuSecs [int] = 319075'

--------------------------------------------------------------------------------------------

EventAgentLogout

=====================

2022-12-14 08:10:09.778 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory : Endpoint 'SERVER1' received message ''EventAgentLogout' (74) attributes:

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

'geo-location-agent' [str] = "CTC"

AttributeEventSequenceNumber [long] = 744889386

TimeStamp:

AttributeTimeinSecs [int] = 1671027009

AttributeTimeinuSecs [int] = 779569'

I have different event log format but I am trying to extract the common fields (highlighted and underlined fields). Since the log format is different I am not sure how to extract the values using single rex or regex query. The field I am looking for is:

Endpoint = SERVER1

received message = EventAgentLogout

AttributeThisDN = 1990613829

AttributeAgentID= 34434343

AttributeAgentWorkMode = AfterCallWork

ReasonCode = Break

There are few fields like "Reasoncode" does not present in few logs events. But the command field is "AttributeThisDN" which will be unique in all the event

Re: How to extract same set of values from different log format?

ITWhisperer — Wed, 14 Dec 2022 17:16:57 GMT

If there is a possibility that the field does not exist in the event, the extraction should be split across multiple commands:

| rex "Endpoint '(?<endpoint>\w+)' received message ''(?<received_message>[^']+)'" | rex "AttributeThisDN[^\"]+\"(?<AttributeThisDN>[^\"]+)\"" | rex "AttributeAgentID[^\"]+\"(?<AttributeAgentID>[^\"]+)\"" | rex "AttributeAgentWorkMode[^\[]+\[[^\[]+\[(?<AttributeAgentWorkMode>[^\]]+)\]" | rex "ReasonCode[^\"]+\"(?<ReasonCode>[^\"]+)\""

topic Re: How to extract same set of values from different log format? in Splunk Search

How to extract same set of values from different log format?

Re: How to extract same set of values from different log format?