Extract fields from the log

mariojost — Mon, 15 Feb 2021 10:38:04 GMT

I am trying to get logs from a firewall into splunk. Usually i work with regex to extract the fields, but these logs dont come in a predictable manner, so the fields are not always there, or in the same order. So what i try to do is let splunk automatically detect the fields for itself. Given following example raw log:

2021-02-15T09:50:22Z %FTD-6-430002: EventPriority: Low, DeviceUUID: ef9c5cce-2400-11eb-a8f2-ce5d579dab29, InstanceID: 5, FirstPacketSecond: 2021-02-15T09:50:22Z, ConnectionID: 26265, AccessControlRuleAction: Allow, SrcIP: 195.180.144.165, DstIP: 172.16.20.86, SrcPort: 49609, DstPort: 443, Protocol: tcp, IngressInterface: INT_DMZ_External, EgressInterface: INT_LAN, IngressZone: DMZ_External, EgressZone: LAN, IngressVRF: Global, EgressVRF: Global, ACPolicy: Merbag Default Access Control Policy, AccessControlRuleName: WAP2LAN, Prefilter Policy: Merbag Default Prefilter Policy, Client: SSL client, ApplicationProtocol: HTTPS, InitiatorPackets: 3, ResponderPackets: 1, InitiatorBytes: 389, ResponderBytes: 70, NAPPolicy: Balanced Security and Connectivity, URLReputation: Unknown, URL: https://adfs.company.com

I have tried lots of things in props.conf but sadly, nothing seems to work.

[firewall] category = Network & Security TIME_FORMAT = %Y-%m-%dT%H:%M:%SZ TZ = Europe/London KV_MODE = auto FIELD_DELIMITER = , HEADER_FIELD_DELIMITER = :

So the first 3 lines work great, but the last 3 lines dont seem to work at all. Sadly, there is not onboard GUI in splunk where i can run the config against a log and see the output live. Its just editing the config file and restarting the service over and over again. It would be helpful if there were actual examples (config & example log) in the documentation.

Re: Extract fields from the log

manjunathmeti — Mon, 15 Feb 2021 11:09:30 GMT

hi @mariojost,
You can extract these fields at search time using transforms.
props.conf

[firewall] KV_MODE=none REPORT-extractfields = mytransform

transforms.conf

[mytransform] REGEX= \:?\s([\s\w]+):\s([^,]+) FORMAT= $1::$2

If this reply helps you, an upvote/like would be appreciated.

topic Re: Extract fields from the log in Getting Data In

Extract fields from the log

Re: Extract fields from the log