All Apps and Add-ons

Is there a way to check the field extraction regexes for a particular sourcetype?

soumyasaha25
Contributor

Is there a way to check the field extraction regexes for a particular sourcetype (say a sourcetype from the TA for AWS - aws:cloudwatchlogs:vpcflow?
I have reasons to believe that a few of the extractions are not working properly and extracting junk/irrelevant data. Is there a way to modify the extractions for the sourcetype of the add on?
I really would prefer not to create a new sourcetype with custom extractions as much as possible.

0 Karma
1 Solution

adonio
Ultra Champion

Yes! there is a way!

download the app (or TA) and navigate to .../default/ directory. open props.conf with your text editor and find your sourcetype [stanza]
in your case - this how it looks like:

[aws:cloudwatchlogs:vpcflow]
SHOULD_LINEMERGE = false
EXTRACT-all=^\s*(\d{4}-\d{2}-\d{2}.\d{2}:\d{2}:\d{2}[.\d\w]*)?\s*(?P<version>[^\s]+)\s+(?P<account_id>[^\s]+)\s+(?P<interface_id>[^\s]+)\s+(?P<src_ip>[^\s]+)\s+(?P<dest_ip>[^\s]+)\s+(?P<src_port>[^\s]+)\s+(?P<dest_port>[^\s]+)\s+(?P<protocol_code>[^\s]+)\s+(?P<packets>[^\s]+)\s+(?P<bytes>[^\s]+)\s+(?P<start_time>[^\s]+)\s+(?P<end_time>[^\s]+)\s+(?P<vpcflow_action>[^\s]+)\s+(?P<log_status>[^\s]+)

EVAL-duration=end_time-start_time
FIELDALIAS-src_ip_as_src = src_ip as src
FIELDALIAS-dest_ip_as_dest = dest_ip as dest

LOOKUP-protocol=vpcflow_protocol_code_lookup protocol_code OUTPUT protocol protocol_full_name
LOOKUP-action=vpcflow_action_lookup vpcflow_action OUTPUT action
REPORT-extract-region = extract-region


# unify account ID field
FIELDALIAS-aws-account-id = account_id as aws_account_id

if you would like to modify, create local directory under the app directory and create a new props.conf file in it.
write your relevant configurations and save. restart or debug refresh and you are all set.

NOTE: pay attention to references to transforms if any. (there are noon in this particular sourcetype) also pay attention to lookups as they are referred in this sourcetype props.

hope it helps

View solution in original post

0 Karma

adonio
Ultra Champion

Yes! there is a way!

download the app (or TA) and navigate to .../default/ directory. open props.conf with your text editor and find your sourcetype [stanza]
in your case - this how it looks like:

[aws:cloudwatchlogs:vpcflow]
SHOULD_LINEMERGE = false
EXTRACT-all=^\s*(\d{4}-\d{2}-\d{2}.\d{2}:\d{2}:\d{2}[.\d\w]*)?\s*(?P<version>[^\s]+)\s+(?P<account_id>[^\s]+)\s+(?P<interface_id>[^\s]+)\s+(?P<src_ip>[^\s]+)\s+(?P<dest_ip>[^\s]+)\s+(?P<src_port>[^\s]+)\s+(?P<dest_port>[^\s]+)\s+(?P<protocol_code>[^\s]+)\s+(?P<packets>[^\s]+)\s+(?P<bytes>[^\s]+)\s+(?P<start_time>[^\s]+)\s+(?P<end_time>[^\s]+)\s+(?P<vpcflow_action>[^\s]+)\s+(?P<log_status>[^\s]+)

EVAL-duration=end_time-start_time
FIELDALIAS-src_ip_as_src = src_ip as src
FIELDALIAS-dest_ip_as_dest = dest_ip as dest

LOOKUP-protocol=vpcflow_protocol_code_lookup protocol_code OUTPUT protocol protocol_full_name
LOOKUP-action=vpcflow_action_lookup vpcflow_action OUTPUT action
REPORT-extract-region = extract-region


# unify account ID field
FIELDALIAS-aws-account-id = account_id as aws_account_id

if you would like to modify, create local directory under the app directory and create a new props.conf file in it.
write your relevant configurations and save. restart or debug refresh and you are all set.

NOTE: pay attention to references to transforms if any. (there are noon in this particular sourcetype) also pay attention to lookups as they are referred in this sourcetype props.

hope it helps

0 Karma

soumyasaha25
Contributor

oh. yes..figured it out. this had completely slipped my mind. Thanks

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...