Extract fields on the Heavy Forwarder

[aalaa]

hello,
Please I need to know how can I extract field in the heavy forwarder?I try by adding the extract stanza in the propos. conf but it doesn't work.

Thank you in advance

[harsmarvania57]

Hi,

You can achieve this extraction at Search time on Search Head, no need to extract fields at Index time on Heavy Forwarder.

Have a look at my answer on https://answers.splunk.com/answers/744449/how-to-parse-out-fields.html , once you configure that settings on Search Head it will automatically extract FACILITY, TYPE and ACTION fields. I have tested this in my lab environment with sample data you have provided and it is extracting all 3 fields.

[harsmarvania57]

Hi,

Can you please provide some sample data(Mask any sensitive data) ? Also provide what you want to extract in new field and your props.conf configuration.

[aalaa]

Sorry @harsmarvania57 i can't provide any type of data just i want to know the steps to extract data in a heavy forwarder
Thank you

[harsmarvania57]

Have a look at https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Configureindex-timefieldextraction for index time field extraction

[aalaa]

@harsmarvania57 thank you for this document ,
for more explication , i need to parsing data on the heavy forwarder before it will be sending to the indexer , so i configure the prop.conf and transforms.conf to extract the "service" field :

Props.conf :
[source::udp:514]
TRANSFORMS-src= index1

Transforms.conf :
[index1]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Personnaliser
pulldown_type = 1
EXTRACT-SERVICE = ^(?:[^>\n]*>){9}(?P\w+)

But it dosen't work .

[harsmarvania57]

EXTRACT- is search time extraction. If you want to extract field at Index time then use below configuration on heavy forwarder.

props.conf

[source::udp:514]
TRANSFORMS-src= service_ext

transforms.conf

[service_ext]
REGEX =  <your regex with correct capturing group>
FORMAT = service::"$<capturing group number>"
WRITE_META = true

As you didn't provide any sample data I can't help with REGEX and due to that can't help with FORMAT as well.

[rajashaey]

@harsmarvania57 
I'm also having same problem. I'm doing it in HF.
Note: My logs has headers. Does it causes the problem though?

Props:
[sourcetype::aws:cloudwatchlogs:vpcflow]
TRANSFORMS-vpc = vpc_flowcustom

Transforms:
[vpc_flowcustom]
REGEX = ^\s*(?P<account_id>[^\s]+)\s+(?P<version>[^\s]+)\s+(?P<interface_id>[^\s]+)\s+(?P<src_ip>[^\s]+)\s+(?P<dest_ip>[^\s]+)\s+(?P<src_port>[^\s]+)\s+(?P<dest_port>[^\s]+)\s+(?P<protocol_code>[^\s]+)\s+(?P<packets>[^\s]+)\s+(?P<bytes>[^\s]+)\s+(?P<start_time>[^\s]+)\s+(?P<action>[^\s]+)\s+(?P<end_time>[^\s]+)\s+(?P<log_status>[^\s]+)\s+(?P<vpc_id>[^\s]+)\s+(?P<subnet_id>[^\s]+)\s+(?P<instance_id>[^\s]+)\s+(?P<tcp_flags>[^\s]+)\s+(?P<type>[^\s]+)\s+(?P<pkt_srcaddr>[^\s]+)\s+(?P<vpc_region>[^\s]+)\s+(?P<pkt_dstaddr>[^\s]+)\s+(?P<az_id>[^\s]+)\s+(?P<sublocation_type>[^\s]+)\s+(?P<sublocation_id>[^\s]+)\s+(?P<flow_direction>[^\s]+)\s+(?P<traffic_path>[^\s]+)\s+(?P<pkt_src_aws_service>[^\s]+)\s+(?P<pkt_dst_aws_service>[^\s]+)
WRITE_META = true