Splunk Enterprise

Dynamic field extraction with regex syslog's?

snix
Communicator

We are ingesting Firepower logs via syslog using the cisco:asa TA. Many of the events I am interested in are Threat Defense events that are tied to an ID like this FTD-6-430002. When I narrow down my search to events with just that ID I find the rest of the event has plenty of info in key:value pairs but no fields have been extracted from the pairs.

Sanitized example event:

Mar 3 16:01:21 172.16.51.72 Mar 03 2023 22:01:21 firepower : %FTD-6-430002: EventPriority: Low, DeviceUUID: 00000-0000-0000-000000000000, InstanceID: 1, FirstPacketSecond: 2023-03-03T22:01:21Z, ConnectionID: 5000, AccessControlRuleAction: Allow, SrcIP: 100.100.100.100, DstIP: 200.200.200.200, SrcPort: 60000, DstPort: 10, Protocol: tcp

Is there a regex command that can dynamically extract all the field names from something like "DstPort: 10" to Field Name of DstPort with a value of 10? 

I know Cisco provides a eStreamer TA that may extract these fields but it looks very involved to setup and I already have the syslog configured. 

Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

The extract command is made for that kind of parsing, but it will be tripped up by the extra ":" separators in the timestamp.

Regex will do the job, however, but there's no special command for it.  Assuming the fields are always in the same order then this will do it.

| rex "firepower : (?<firepower>[^:]+): EventPriority: (?<EventPriority>\w+), DeviceUUID: (?<DeviceUUID>[^,]+), InstanceID: (?<InstanceID>[^,]+), FirstPacketSecond: (?<FirstPacketSecond>[^,]+), ConnectionID: (?<ConnectionID>[^,]+), AccessControlRuleAction: (?<AccessControlRuleAction>[^,]+), SrcIP: (?<SrcIP>[^,]+), DstIP: (?<DstIP>[^,]+), SrcPort: (?<SrcPort>\d+), DstPort: (?<DstPort>\d+), Protocol: \w+"

If the fields might appear in any order then a separate rex command is needed for each one.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

The extract command is made for that kind of parsing, but it will be tripped up by the extra ":" separators in the timestamp.

Regex will do the job, however, but there's no special command for it.  Assuming the fields are always in the same order then this will do it.

| rex "firepower : (?<firepower>[^:]+): EventPriority: (?<EventPriority>\w+), DeviceUUID: (?<DeviceUUID>[^,]+), InstanceID: (?<InstanceID>[^,]+), FirstPacketSecond: (?<FirstPacketSecond>[^,]+), ConnectionID: (?<ConnectionID>[^,]+), AccessControlRuleAction: (?<AccessControlRuleAction>[^,]+), SrcIP: (?<SrcIP>[^,]+), DstIP: (?<DstIP>[^,]+), SrcPort: (?<SrcPort>\d+), DstPort: (?<DstPort>\d+), Protocol: \w+"

If the fields might appear in any order then a separate rex command is needed for each one.

---
If this reply helps you, Karma would be appreciated.

snix
Communicator

Thank you! Yeah I was hoping there was a way to just pull the filed name from the event automatically but for what I need it for right now I just need a few fields. 

Yeah the events seem to vary a bit on what fields are included and not included so I opted to take your second suggestion and spit them up and that worked like a charm:

 

| rex "AccessControlRuleAction: (?<AccessControlRuleAction>[^,]+)" 
| rex "SrcIP: (?<SrcIP>[^,]+)" 
| rex "DstIP: (?<DstIP>[^,]+)" 
| rex "DstPort: (?<DstPort>[^,]+)" 

 

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...