Splunk Enterprise

How do I exclude 5th column from a CSV file using props.conf and tranforms.conf?

super_saiyan
Communicator

Can someone please guide me how do i exclude 5th column from a CSV file using props.conf and tranform.conf 

Let's assume i have 100 column in my csv file.

Labels (1)
0 Karma

PickleRick
SplunkTrust
SplunkTrust

If you want to filter the column from indexing, you need to simply use a SEDCMD within your props.conf and that's it. The tricky part might be working out the proper regex considering that csv files can usually have quoted strings including commas and quoted quotes which makes acoounting for all possibilities... annoying.

For a start, _not_ taking into account all those cases, just assuming that you have simple values between commas, you need something like that in props.conf stanza for your source/sourcetype/host:

SEDCMD-cutcolumn = s/^(([^,]*,){4})[^,]*,/\1/

 Might not work properly with less than 6 solumns present in the input

gcusello
SplunkTrust
SplunkTrust

Hi @super_saiyan,

are you speaking of exclusion from indexing (at index time) or from result displaying (at search time)?

Ciao.

Giuseppe

super_saiyan
Communicator

Thanks for the quick response @gcusello 

yes, I'm referring to the exclusion from indexing.

could you please help

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @super_saiyan,

as hinted by @PickleRick, you have to use the SEDCMD command or use props and transforms associated to the sourcetype you're using.

In few words, you have to find the regex to identify the column to exclude, e.g. if you have 100 columns divided by comma ",", you could use a regex like this:

in props.conf

[your_sourcetype]
TRANSFORMS-delete_column_80 = delete_column_80

in transforms.conf

[delete_column_80]
REGEX = ^(([^,]+,){80})[^,]+,(([^,]+,){19})
FORMAT = $1$2
DEST_KEY = _raw

For more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/Anonymizedata

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

Take Your Breath Away with Splunk Risk-Based Alerting (RBA)

WATCH NOW!The Splunk Guide to Risk-Based Alerting is here to empower your SOC like never before. Join Haylee ...

SignalFlow: What? Why? How?

What is SignalFlow? Splunk Observability Cloud’s analytics engine, SignalFlow, opens up a world of in-depth ...

Federated Search for Amazon S3 | Key Use Cases to Streamline Compliance Workflows

Modern business operations are supported by data compliance. As regulations evolve, organizations must ...