Multiple Value Field Extraction Help

[jaoui]

I am trying to come up with a Regex that will extract several field values from an event which can potentially have several occurrences of each field in a given event:

One of the Regexes that I am working on so far is something like this:

... | rex "Hex-STRING:\s(?<macstatus>([a-fA-F0-9][a-fA-F0-9]){1}?)"

But i can't figure out how to account for multiple instances of the variable

Sample events:

2010-10-20 08:39:21 switch.local [UDP: [192.168.1.6]:54862->[192.168.1.16]]: Trap , DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (977272346) 113 days, 2:38:43.46, SNMPv2-MIB::snmpTrapOID.0 = OID: CISCO-MAC-NOTIFICATION-MIB::cmnMacChangedNotification, CISCO-MAC-NOTIFICATION-MIB::cmnHistMacChangedMsg.24 = Hex-STRING: <font color=22345>02 01 0F D4 9A 20 ED C6 7E 00 30 02 00 3C 00 23 32 2D E4 C0 01 F8 02 03 06 00 23 32 2D E4 C0 01 F8 00</font> , CISCO-MAC-NOTIFICATION-MIB::cmnHistTimestamp.24 = Wrong Type (should be Timeticks): INTEGER: 977272346
2010-10-20 08:40:21 switch.local [UDP: [192.168.1.6]:62723->[192.168.1.16]]: Trap , DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (977278346) 113 days, 2:39:43.46, SNMPv2-MIB::snmpTrapOID.0 = OID: CISCO-MAC-NOTIFICATION-MIB::cmnMacChangedNotification, CISCO-MAC-NOTIFICATION-MIB::cmnHistMacChangedMsg.25 = Hex-STRING: 01 01 0F D4 9A 20 ED C6 7E 00 30 00 , CISCO-MAC-NOTIFICATION-MIB::cmnHistTimestamp.25 = Wrong Type (should be Timeticks): INTEGER: 977278346

The string of Hex characters contains the information I am looking for.
Every 11 octets of Hex represents one group.

For example, this would be one group (the final octet of '00' in each event marks the end of the string):

01 01 0F D4 9A 20 ED C6 7E 00 30

Each group consists of four fields.

1. mac_status = Octet #1
   • Ex: (01)
     
     
2. mac_vlan = Octet #2,3
   • Ex: (01 0F)
     
     
3. mac_addr = Octet #4,5,6,7,8,9
   • Ex: (D4 9A 20 ED C6 7E)
     
     
4. mac_portid = Octet #10,11
   • Ex: (00 30)

Thank you, David

[mikaelbje]

7 years after:

This one fixes an issue with the Hex STRING being split by a newline. Documenting this in case it is of value to someone.

| rex max_match=50000 field=_raw "(?msi)Hex-STRING:\s?(?<Hex_STRING>.+)\t"
| rex field=Hex_STRING mode=sed "s/\s\n/ /g"
| rex max_match=50000 field=Hex_STRING "(?<mac_status>([a-fA-F0-9][a-fA-F0-9]){1}) (?<mac_vlan>[a-fA-F0-9]{2} [a-fA-F0-9]{2}) (?<mac_addr>[a-fA-F0-9]{2} [a-fA-F0-9]{2} [a-fA-F0-9]{2} [a-fA-F0-9]{2} [a-fA-F0-9]{2} [a-fA-F0-9]{2}) (?<mac_portid>[a-fA-F0-9]{2} [a-fA-F0-9]{2})"
| rex field=mac_vlan mode=sed "s/ //g"
| rex field=mac_portid mode=sed "s/ //g"
| rex field=mac_addr mode=sed "s/ /:/g"
| eval mac_addr=lower(mac_addr)
| stats latest(_time) AS _time values(host) AS host latest(mac_status) AS mac_status latest(mac_vlan) AS vlan latest(mac_portid) AS mac_portid BY mac_addr
| eval vlan=tonumber(vlan,16)
| eval mac_portid=tonumber(mac_portid,16)
| eval mac_status=tonumber(mac_status,16)

[Ledion_Bitincka · Accepted solution]

You can have multiple capturing groups in your regex. Is this what you're looking for?

... | rex "Hex-STRING: (?<mac_status>([a-fA-F0-9]{2}) (?<mac_vlan>[a-fA-F0-9]{2} [a-fA-F0-9]{2}) (?<mac_addr>[a-fA-F0-9]{2} [a-fA-F0-9]{2} [a-fA-F0-9]{2} [a-fA-F0-9]{2} [a-fA-F0-9]{2} [a-fA-F0-9]{2}) (?<mac_portid>[a-fA-F0-9]{2} [a-fA-F0-9]{2}) 00"