topic Re: How to extract fields in Heavy Forwarder? in Splunk Dev

How to extract fields in Heavy Forwarder?

R_M — Thu, 31 Mar 2022 16:30:57 GMT

Guys if you help me to extract fields from the raw events in props.conf in HF, I tried EXTRACT command seems my regex is not ok or not sure what is the issue.

I want to extract field and give name to them.

Regex I tried:

^(?:[^,\n]*,){7}(?<src_ip>[^,]+),(?<dst_ip>[^,]+)(?:[^:\n]*:){2}\d+,\d+,\d+,(?<src_port>\d+),(?<dst_port>\d+)(?:[^,\n]*,){5}(?<action>[^,]+)(?:[^,\n]*,){38}

Also,

^(?:[^,\n]*,){7}src_ip=(?<src_ip>[^,]+),dst_ip=(?<dst_ip>[^,]+)(?:[^:\n]*:){2}\d+,\d+,\d+,src_port=(?<src_port>\d+),dst_port=(?<dst_port>\d+)(?:[^,\n]*,){5}action=(?<action>[^,]+)(?:[^,\n]*,){38}

Sample log:

Mar 31 18:18:35 LUM-EVERE-PAFW-R8-17-T1 1,2022/03/31 18:18:35,015701001564,TRAFFIC,drop,2305,2022/03/31 18:18:35,10.81.13.68,34.240.162.53,0.0.0.0,0.0.0.0,prodedfl_access_1289,,,not-applicable,vsys4,prodedfl,prodcore,ae1.1512,,Syslog_Server,2022/03/31 18:18:35,0,1,60353,443,0,0,0x0,tcp,deny,66,66,0,1,2022/03/31 18:18:35,0,any,0,7022483376390954281,0x8000000000000000,10.0.0.0-10.255.255.255,Ireland,0,1,0,policy-deny,920,0,0,0,Production,LUM-EVERE-PAFW-R8-17-T1,from-policy,,,0,,0,,N/A,0,0,0,0,2d8c02f8-e86f-43cf-a459-01acdb26580a,0,0,,,,,,,

Please help me to extract fields like src_ip, dst_ip, src_port, dst_port, action etc.

Re: How to extract fields in Heavy Forwarder?

richgalloway — Thu, 31 Mar 2022 17:01:10 GMT

Is that a single event or multiple events?

Please identify the fields within the log (which IP address is source/dest, etc.).

Re: How to extract fields in Heavy Forwarder?

mayurr98 — Thu, 31 Mar 2022 17:13:14 GMT

Hey could you please try this :

| makeresults | eval _raw="Mar 31 18:18:35 LUM-EVERE-PAFW-R8-17-T1 1,2022/03/31 18:18:35,015701001564,TRAFFIC,drop,2305,2022/03/31 18:18:35,10.81.13.68,34.240.162.53,0.0.0.0,0.0.0.0,prodedfl_access_1289,,,not-applicable,vsys4,prodedfl,prodcore,ae1.1512,,Syslog_Server,2022/03/31 18:18:35,0,1,60353,443,0,0,0x0,tcp,deny,66,66,0,1,2022/03/31 18:18:35,0,any,0,7022483376390954281,0x8000000000000000,10.0.0.0-10.255.255.255,Ireland,0,1,0,policy-deny,920,0,0,0,Production,LUM-EVERE-PAFW-R8-17-T1,from-policy,,,0,,0,,N/A,0,0,0,0,2d8c02f8-e86f-43cf-a459-01acdb26580a,0,0,,,,,,," | rex "\d{2}:\d{2}:\d{2},(?<src_ip>(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))\,(?<dst_ip>(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))\,(?:[^:\n]*:){2}\d+,\d+,\d+,(?<src_port>\d+)\,(?<dst_port>[^\,]+)(?:[^,\n]*,){5}(?<action>[^\,]+)"

let me know if this helps!

Thanks,

Mayur

Re: How to extract fields in Heavy Forwarder?

R_M — Thu, 31 Mar 2022 17:33:50 GMT

Can I apply this regex in props.conf under EXTRACT commmand, also I also want to give name to the field, example: src_ip: 10.82.69.5

Re: How to extract fields in Heavy Forwarder?

mayurr98 — Thu, 31 Mar 2022 17:55:31 GMT

yes you can use this regex in props.conf.

if you want to add a search time field extraction within props.conf, just use EXTRACT

[your-sourcetype] EXTRACT-<class> = [<regex>|<regex> in <src_field>] * Used to create extracted fields (search-time field extractions) that do not reference transforms.conf stanzas.

for reference see : http://docs.splunk.com/Documentation/Splunk/8.2.5/Admin/Propsconf

Please keep in mind that this will require a refresh/debug= http[s]://[splunkweb hostname]:[splunkweb port]/debug/refresh

Re: How to extract fields in Heavy Forwarder?

R_M — Fri, 01 Apr 2022 05:59:25 GMT

Please help me how I can apply it, if you can share command

Re: How to extract fields in Heavy Forwarder?

PickleRick — Fri, 01 Apr 2022 06:24:13 GMT

Wait a minute. EXTRACT is for search-time extractions. You don't use it on HF since you don't search on HF.

Re: How to extract fields in Heavy Forwarder?

mayurr98 — Fri, 01 Apr 2022 14:51:38 GMT

you need to use this regex on search head

go to

Settings » Fields » Field extractions » Add new

Destination App: <your_app> Name: <name> Apply to: choose sourcetype : named <your_sourcetype> Type: Inline Extraction/Transform: \d{2}:\d{2}:\d{2},(?<src_ip>(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))\,(?<dst_ip>(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?))\,(?:[^:\n]*:){2}\d+,\d+,\d+,(?<src_port>\d+)\,(?<dst_port>[^\,]+)(?:[^,\n]*,){5}(?<action>[^\,]+)

let me know if this helps!