How to extract same set of values from different log format?

[ravir_jbp]

EventAgentLogin

==================

2022-12-14 06:39:03.875 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory     : Endpoint 'SERVER1' received message ''EventAgentLogin' (73) attributes:

AttributeReasons [bstr] = KVList:

              'referrer' [str] = "https://aaa.test.com"

              'clientApp' [str] = "asdfsfdf"

              'location' [str] = "server"

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

              'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0077MP"

              'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744434548

TimeStamp:

              AttributeTimeinSecs [int] = 1671021543

              AttributeTimeinuSecs [int] = 882837'

-------------------------------------------------------------------------------------

agent ready state"

=======================

2022-12-14 07:59:12.764 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory     : Endpoint 'SERVER1' received message ''EventAgentReady' (75) attributes:

AttributeReasons [bstr] = KVList:

              'site' [str] = "CTC"

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

              'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

              'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744780812

TimeStamp:

              AttributeTimeinSecs [int] = 1671026352

              AttributeTimeinuSecs [int] = 766178'

--------------------------------------------------------

EventAgentNotReady

==================

2022-12-14 08:01:31.602 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory     : Endpoint 'SERVER1' received message ''EventAgentNotReady' (76) attributes:

AttributeReasons [bstr] = KVList:

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

              'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

              'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744808316

TimeStamp:

----------------------------------------------------------

Training

==========

2022-12-14 08:02:47.211 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory     : Endpoint 'SERVER1' received message ''EventAgentNotReady' (76) attributes:

AttributeReasons [bstr] = KVList:

              'ReasonCode' [str] = "Training"

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

              'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

              'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744821504

TimeStamp:

              AttributeTimeinSecs [int] = 1671026567

              AttributeTimeinuSecs [int] = 209306'

-------------------------------------------------------------------------

"EventAgentNotReady" AND "Break"

====================================

2022-12-14 08:04:34.025 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory     : Endpoint 'SERVER1'  received message ''EventAgentNotReady' (76) attributes:

AttributeReasons [bstr] = KVList:

              'ReasonCode' [str] = "Break"

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

              'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

              'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 0 [Unknown]

AttributeEventSequenceNumber [long] = 744838251

TimeStamp:

              AttributeTimeinSecs [int] = 1671026674

              AttributeTimeinuSecs [int] = 24284'

----------------------------------------------------------------------------------

AfterCallWork

===============

2022-12-14 08:07:31.310 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory     : Endpoint 'SERVER1'  received message ''EventAgentNotReady' (76) attributes:

AttributeReasons [bstr] = KVList:

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

              'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

              'ReasonCode' [str] = "ManualSetACWPeriod"

              'WrapUpTime' [str] = "untimed"

              'geo-location-agent' [str] = "CTC"

AttributeAgentWorkMode [int] = 3 [AfterCallWork]

AttributeEventSequenceNumber [long] = 744864731

TimeStamp:

              AttributeTimeinSecs [int] = 1671026851

              AttributeTimeinuSecs [int] = 319075'

--------------------------------------------------------------------------------------------

EventAgentLogout

=====================

2022-12-14 08:10:09.778 TRACE 12632 --- [New I/O client worker #1-6] c.i.e.g.workflows.ServerEventFactory     : Endpoint 'SERVER1'  received message ''EventAgentLogout' (74) attributes:

AttributeThisDN [str] = "1990613829"

AttributeAgentID [str] = "34434343"

AttributeExtensions [bstr] = KVList:

              'AgentSessionID' [str] = "027DK7N0IC9H5D7NF885C2LAES0078C6"

              'geo-location-agent' [str] = "CTC"

AttributeEventSequenceNumber [long] = 744889386

TimeStamp:

              AttributeTimeinSecs [int] = 1671027009

              AttributeTimeinuSecs [int] = 779569'

I have different event log format but I am trying to extract the common fields (highlighted and underlined fields). Since the  log format is different I am not sure how to extract the values using single rex or regex query. The field I am looking for is:

Endpoint = SERVER1

received message = EventAgentLogout

AttributeThisDN = 1990613829

AttributeAgentID= 34434343

AttributeAgentWorkMode = AfterCallWork

ReasonCode = Break

There are few fields like "Reasoncode" does not present in few logs events. But the command field is "AttributeThisDN" which will be unique in all the event

[ITWhisperer]

If there is a possibility that the field does not exist in the event, the extraction should be split across multiple commands:

| rex "Endpoint '(?<endpoint>\w+)' received message ''(?<received_message>[^']+)'"
| rex "AttributeThisDN[^\"]+\"(?<AttributeThisDN>[^\"]+)\""
| rex "AttributeAgentID[^\"]+\"(?<AttributeAgentID>[^\"]+)\""
| rex "AttributeAgentWorkMode[^\[]+\[[^\[]+\[(?<AttributeAgentWorkMode>[^\]]+)\]"
| rex "ReasonCode[^\"]+\"(?<ReasonCode>[^\"]+)\""