topic How to edit my props and transforms to filter out certain phrases in ASA logs? in Getting Data In

How to edit my props and transforms to filter out certain phrases in ASA logs?

jlarkins — Thu, 14 Jun 2018 16:34:16 GMT

Hi,

I've read a few articles on filtering data inputs. Basically I have a noisy ASA that I'm logging, and I want to discard anything with these phrases in them. They are eating up my licensing quota and I don't need them.

Built outbound TCP connection
Teardown TCP connection
Teardown UDP connection
Built dynamic TCP translation
Teardown dynamic TCP translation

I have the following set up for my configuration files but it's not working. Can someone help me out, I'm not good at regex 😕
transforms.conf

[geo_us_states]
external_type = geo
filename = geo_us_states.kmz

[geo_countries]
external_type = geo
filename = geo_countries.kmz

[geo_attr_us_states]
filename = geo_attr_us_states.csv

[geo_attr_countries]
filename = geo_attr_countries.csv

[discard_events]
REGEX = Built outbound TCP connection
DEST_KEY = queue
FORMAT = nullQueue
[discard_events]
REGEX = Teardown dynamic TCP translation
DEST_KEY = queue
FORMAT = nullQueue
[discard_events]
REGEX = Teardown TCP connection
DEST_KEY = queue
FORMAT = nullQueue
[discard_events]
REGEX = Teardown UDP connection
DEST_KEY = queue
FORMAT = nullQueue
[discard_events]
REGEX = Built dynamic TCP translation
DEST_KEY = queue
FORMAT = nullQueue

props.conf

#   Version 7.1.1
[sendmail]
FIELDALIAS-msgid = mid AS msgid
EXTRACT-qid = sendmail[^\]]*\]:\s+(?[^:]*):
 [source::/var/log/foo]
 # Transforms must be applied in this order
 # to make sure events are dropped on the
 # floor prior to making their way to the
 # index processor
TRANSFORMS-set = discard_events

Re: How to edit my props and transforms to filter out certain phrases in ASA logs?

lacastillo — Tue, 29 Sep 2020 20:03:49 GMT

In props.conf, each of your TRANSFORMS-set values must be listed and comma separated

ex:
TRANSFORMS-set = discard_events, discard_events1, discard_events2, discard_events3, discard_events4

Then go to your transforms.conf and make sure each stanza correctly corresponds to each value in your TRANSFORMS-set from props.conf

transforms.conf stanza ex:
[discard_events]
REGEX = Built outbound TCP connection
DEST_KEY = queue
FORMAT = nullQueue

[discard_events1]
REGEX = Teardown dynamic TCP translation
DEST_KEY = queue
FORMAT = nullQueue

[discard_events2]
REGEX = Teardown TCP connection
DEST_KEY = queue
FORMAT = nullQueue

[discard_events3]
REGEX = Teardown UDP connection
DEST_KEY = queue
FORMAT = nullQueue

[discard_events4]
REGEX = Built dynamic TCP translation
DEST_KEY = queue
FORMAT = nullQueue

Let me know if that helped!

Re: How to edit my props and transforms to filter out certain phrases in ASA logs?

jlarkins — Tue, 29 Sep 2020 20:04:49 GMT

I did that and it still wasn't dropping the data then I put the TRANSFORMS-set line at the top of the props.conf line instead of at the bottom and that fixed it! Thanks a bunch

P.s. for future viewers the files to edit are these for the ASA addon:
etc\apps\Splunk_TA_cisco-asa\default

Re: How to edit my props and transforms to filter out certain phrases in ASA logs?

RHASQaL — Tue, 29 Sep 2020 20:05:35 GMT

You shouldn't edit the files in the default folders, as any changes you make may be overwritten when you install a new version of Splunk or a new version of an app. You should create the configuration file in the corresponding local folder (if one doesn't already exist) and add any changes there. The local folders should remain unaffected when installing new versions of apps or Splunk.

How to manage configuration files is covered in the Admin documentation here(http://docs.splunk.com/Documentation/Splunk/7.1.1/Admin/Aboutconfigurationfiles).

One of the problems with your original transforms.conf was that you reused the stanza name [discard_events], the stanza names need to be unique. I suspect you ended up with just the last of the [discard_events] being in place. There is a utility called btool (http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurations) that among other things allows you to query what Splunk thinks is defined in the configuration files.

Enjoy your journey with Splunk

For some reason the hyperlinks aren't working, so I've placed the links in brackets.

Re: How to edit my props and transforms to filter out certain phrases in ASA logs?

RHASQaL — Tue, 19 Jun 2018 09:12:45 GMT

Hi
Splunk best practices advise that you don’t edit any files in the default folder, but to create a new configuration file in the corresponding local folder, if one doesn’t already exist. The problem with making changes to the default folder is that any changes may be overwritten when a new version of Splunk, or a new version of the app is installed. Configuration file maintenance is covered in the Admin documentation here - http://docs.splunk.com/Documentation/Splunk/7.1.1/Admin/Aboutconfigurationfiles

One problem with your original transforms.conf is the duplication of the [discard_events] stanza, the stanza names need to be unique. I suspect you ended up with just the final occurrence of the stanza being recognised by Splunk. There is a command line utility called btool - http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/Usebtooltotroubleshootconfigurations - that, amongst other things, allows you to troubleshoot configuration file issues or see what values are being used by your Splunk Enterprise installation.

Enjoy your Splunk journey.