Extract fields from the log

[mariojost]

I am trying to get logs from a firewall into splunk. Usually i work with regex to extract the fields, but these logs dont come in a predictable manner, so the fields are not always there, or in the same order. So what i try to do is let splunk automatically detect the fields for itself. Given following example raw log:

2021-02-15T09:50:22Z %FTD-6-430002: EventPriority: Low, DeviceUUID: ef9c5cce-2400-11eb-a8f2-ce5d579dab29, InstanceID: 5, FirstPacketSecond: 2021-02-15T09:50:22Z, ConnectionID: 26265, AccessControlRuleAction: Allow, SrcIP: 195.180.144.165, DstIP: 172.16.20.86, SrcPort: 49609, DstPort: 443, Protocol: tcp, IngressInterface: INT_DMZ_External, EgressInterface: INT_LAN, IngressZone: DMZ_External, EgressZone: LAN, IngressVRF: Global, EgressVRF: Global, ACPolicy: Merbag Default Access Control Policy, AccessControlRuleName: WAP2LAN, Prefilter Policy: Merbag Default Prefilter Policy, Client: SSL client, ApplicationProtocol: HTTPS, InitiatorPackets: 3, ResponderPackets: 1, InitiatorBytes: 389, ResponderBytes: 70, NAPPolicy: Balanced Security and Connectivity, URLReputation: Unknown, URL: https://adfs.company.com 

I have tried lots of things in props.conf but sadly, nothing seems to work.

[firewall]
category = Network & Security
TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ
TZ = Europe/London
KV_MODE = auto
FIELD_DELIMITER = ,
HEADER_FIELD_DELIMITER = :

So the first 3 lines work great, but the last 3 lines dont seem to work at all. Sadly, there is not onboard GUI in splunk where i can run the config against a log and see the output live. Its just editing the config file and restarting the service over and over again. It would be helpful if there were actual examples (config & example log) in the documentation.

[manjunathmeti · Accepted solution]

hi @mariojost,
You can extract these fields at search time using transforms.
props.conf

[firewall]
KV_MODE=none
REPORT-extractfields = mytransform

transforms.conf

[mytransform]
REGEX= \:?\s([\s\w]+):\s([^,]+)
FORMAT= $1::$2

If this reply helps you, an upvote/like would be appreciated.