topic Re: props.conf in Splunk Enterprise

How do I exclude 5th column from a CSV file using props.conf and tranforms.conf?

super_saiyan — Wed, 18 May 2022 16:26:45 GMT

Can someone please guide me how do i exclude 5th column from a CSV file using props.conf and tranform.conf

Let's assume i have 100 column in my csv file.

Re: props.conf

gcusello — Wed, 18 May 2022 06:59:41 GMT

Hi @super_saiyan,

are you speaking of exclusion from indexing (at index time) or from result displaying (at search time)?

Ciao.

Giuseppe

Re: props.conf

PickleRick — Wed, 18 May 2022 07:11:03 GMT

If you want to filter the column from indexing, you need to simply use a SEDCMD within your props.conf and that's it. The tricky part might be working out the proper regex considering that csv files can usually have quoted strings including commas and quoted quotes which makes acoounting for all possibilities... annoying.

For a start, _not_ taking into account all those cases, just assuming that you have simple values between commas, you need something like that in props.conf stanza for your source/sourcetype/host:

SEDCMD-cutcolumn = s/^(([^,]*,){4})[^,]*,/\1/

Might not work properly with less than 6 solumns present in the input

Re: props.conf

super_saiyan — Wed, 18 May 2022 09:21:05 GMT

Thanks for the quick response @gcusello

yes, I'm referring to the exclusion from indexing.

could you please help

Re: props.conf

gcusello — Wed, 18 May 2022 09:33:49 GMT

Hi @super_saiyan,

as hinted by @PickleRick, you have to use the SEDCMD command or use props and transforms associated to the sourcetype you're using.

In few words, you have to find the regex to identify the column to exclude, e.g. if you have 100 columns divided by comma ",", you could use a regex like this:

in props.conf

[your_sourcetype] TRANSFORMS-delete_column_80 = delete_column_80

in transforms.conf

[delete_column_80] REGEX = ^(([^,]+,){80})[^,]+,(([^,]+,){19}) FORMAT = $1$2 DEST_KEY = _raw

For more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/Anonymizedata

Ciao.

Giuseppe