All Apps and Add-ons

Is there a way to check the field extraction regexes for a particular sourcetype?

soumyasaha25
Contributor

Is there a way to check the field extraction regexes for a particular sourcetype (say a sourcetype from the TA for AWS - aws:cloudwatchlogs:vpcflow?
I have reasons to believe that a few of the extractions are not working properly and extracting junk/irrelevant data. Is there a way to modify the extractions for the sourcetype of the add on?
I really would prefer not to create a new sourcetype with custom extractions as much as possible.

0 Karma
1 Solution

adonio
Ultra Champion

Yes! there is a way!

download the app (or TA) and navigate to .../default/ directory. open props.conf with your text editor and find your sourcetype [stanza]
in your case - this how it looks like:

[aws:cloudwatchlogs:vpcflow]
SHOULD_LINEMERGE = false
EXTRACT-all=^\s*(\d{4}-\d{2}-\d{2}.\d{2}:\d{2}:\d{2}[.\d\w]*)?\s*(?P<version>[^\s]+)\s+(?P<account_id>[^\s]+)\s+(?P<interface_id>[^\s]+)\s+(?P<src_ip>[^\s]+)\s+(?P<dest_ip>[^\s]+)\s+(?P<src_port>[^\s]+)\s+(?P<dest_port>[^\s]+)\s+(?P<protocol_code>[^\s]+)\s+(?P<packets>[^\s]+)\s+(?P<bytes>[^\s]+)\s+(?P<start_time>[^\s]+)\s+(?P<end_time>[^\s]+)\s+(?P<vpcflow_action>[^\s]+)\s+(?P<log_status>[^\s]+)

EVAL-duration=end_time-start_time
FIELDALIAS-src_ip_as_src = src_ip as src
FIELDALIAS-dest_ip_as_dest = dest_ip as dest

LOOKUP-protocol=vpcflow_protocol_code_lookup protocol_code OUTPUT protocol protocol_full_name
LOOKUP-action=vpcflow_action_lookup vpcflow_action OUTPUT action
REPORT-extract-region = extract-region


# unify account ID field
FIELDALIAS-aws-account-id = account_id as aws_account_id

if you would like to modify, create local directory under the app directory and create a new props.conf file in it.
write your relevant configurations and save. restart or debug refresh and you are all set.

NOTE: pay attention to references to transforms if any. (there are noon in this particular sourcetype) also pay attention to lookups as they are referred in this sourcetype props.

hope it helps

View solution in original post

0 Karma

adonio
Ultra Champion

Yes! there is a way!

download the app (or TA) and navigate to .../default/ directory. open props.conf with your text editor and find your sourcetype [stanza]
in your case - this how it looks like:

[aws:cloudwatchlogs:vpcflow]
SHOULD_LINEMERGE = false
EXTRACT-all=^\s*(\d{4}-\d{2}-\d{2}.\d{2}:\d{2}:\d{2}[.\d\w]*)?\s*(?P<version>[^\s]+)\s+(?P<account_id>[^\s]+)\s+(?P<interface_id>[^\s]+)\s+(?P<src_ip>[^\s]+)\s+(?P<dest_ip>[^\s]+)\s+(?P<src_port>[^\s]+)\s+(?P<dest_port>[^\s]+)\s+(?P<protocol_code>[^\s]+)\s+(?P<packets>[^\s]+)\s+(?P<bytes>[^\s]+)\s+(?P<start_time>[^\s]+)\s+(?P<end_time>[^\s]+)\s+(?P<vpcflow_action>[^\s]+)\s+(?P<log_status>[^\s]+)

EVAL-duration=end_time-start_time
FIELDALIAS-src_ip_as_src = src_ip as src
FIELDALIAS-dest_ip_as_dest = dest_ip as dest

LOOKUP-protocol=vpcflow_protocol_code_lookup protocol_code OUTPUT protocol protocol_full_name
LOOKUP-action=vpcflow_action_lookup vpcflow_action OUTPUT action
REPORT-extract-region = extract-region


# unify account ID field
FIELDALIAS-aws-account-id = account_id as aws_account_id

if you would like to modify, create local directory under the app directory and create a new props.conf file in it.
write your relevant configurations and save. restart or debug refresh and you are all set.

NOTE: pay attention to references to transforms if any. (there are noon in this particular sourcetype) also pay attention to lookups as they are referred in this sourcetype props.

hope it helps

0 Karma

soumyasaha25
Contributor

oh. yes..figured it out. this had completely slipped my mind. Thanks

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Community Content Calendar, September edition

Welcome to another insightful post from our Community Content Calendar! We're thrilled to continue bringing ...

Splunkbase Unveils New App Listing Management Public Preview

Splunkbase Unveils New App Listing Management Public PreviewWe're thrilled to announce the public preview of ...

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...