hello,
Please I need to know how can I extract field in the heavy forwarder?I try by adding the extract stanza in the propos. conf but it doesn't work.
Thank you in advance
Hi,
You can achieve this extraction at Search time on Search Head, no need to extract fields at Index time on Heavy Forwarder.
Have a look at my answer on https://answers.splunk.com/answers/744449/how-to-parse-out-fields.html , once you configure that settings on Search Head it will automatically extract FACILITY, TYPE and ACTION fields. I have tested this in my lab environment with sample data you have provided and it is extracting all 3 fields.
Hi,
Can you please provide some sample data(Mask any sensitive data) ? Also provide what you want to extract in new field and your props.conf configuration.
Sorry @harsmarvania57 i can't provide any type of data just i want to know the steps to extract data in a heavy forwarder
Thank you
Have a look at https://docs.splunk.com/Documentation/Splunk/7.3.1/Data/Configureindex-timefieldextraction for index time field extraction
@harsmarvania57 thank you for this document ,
for more explication , i need to parsing data on the heavy forwarder before it will be sending to the indexer , so i configure the prop.conf and transforms.conf to extract the "service" field :
Props.conf :
[source::udp:514]
TRANSFORMS-src= index1
Transforms.conf :
[index1]
DATETIME_CONFIG =
NO_BINARY_CHECK = true
category = Personnaliser
pulldown_type = 1
EXTRACT-SERVICE = ^(?:[^>\n]*>){9}(?P\w+)
But it dosen't work .
EXTRACT- is search time extraction. If you want to extract field at Index time then use below configuration on heavy forwarder.
props.conf
[source::udp:514]
TRANSFORMS-src= service_ext
transforms.conf
[service_ext]
REGEX = <your regex with correct capturing group>
FORMAT = service::"$<capturing group number>"
WRITE_META = true
As you didn't provide any sample data I can't help with REGEX and due to that can't help with FORMAT
as well.
@harsmarvania57
I'm also having same problem. I'm doing it in HF.
Note: My logs has headers. Does it causes the problem though?
Props:
[sourcetype::aws:cloudwatchlogs:vpcflow]
TRANSFORMS-vpc = vpc_flowcustom
Transforms:
[vpc_flowcustom]
REGEX = ^\s*(?P<account_id>[^\s]+)\s+(?P<version>[^\s]+)\s+(?P<interface_id>[^\s]+)\s+(?P<src_ip>[^\s]+)\s+(?P<dest_ip>[^\s]+)\s+(?P<src_port>[^\s]+)\s+(?P<dest_port>[^\s]+)\s+(?P<protocol_code>[^\s]+)\s+(?P<packets>[^\s]+)\s+(?P<bytes>[^\s]+)\s+(?P<start_time>[^\s]+)\s+(?P<action>[^\s]+)\s+(?P<end_time>[^\s]+)\s+(?P<log_status>[^\s]+)\s+(?P<vpc_id>[^\s]+)\s+(?P<subnet_id>[^\s]+)\s+(?P<instance_id>[^\s]+)\s+(?P<tcp_flags>[^\s]+)\s+(?P<type>[^\s]+)\s+(?P<pkt_srcaddr>[^\s]+)\s+(?P<vpc_region>[^\s]+)\s+(?P<pkt_dstaddr>[^\s]+)\s+(?P<az_id>[^\s]+)\s+(?P<sublocation_type>[^\s]+)\s+(?P<sublocation_id>[^\s]+)\s+(?P<flow_direction>[^\s]+)\s+(?P<traffic_path>[^\s]+)\s+(?P<pkt_src_aws_service>[^\s]+)\s+(?P<pkt_dst_aws_service>[^\s]+)
WRITE_META = true