How do I exclude 5th column from a CSV file using props.conf and tranforms.conf?

[super_saiyan]

Can someone please guide me how do i exclude 5th column from a CSV file using props.conf and tranform.conf 

Let's assume i have 100 column in my csv file.

[PickleRick]

If you want to filter the column from indexing, you need to simply use a SEDCMD within your props.conf and that's it. The tricky part might be working out the proper regex considering that csv files can usually have quoted strings including commas and quoted quotes which makes acoounting for all possibilities... annoying.

For a start, _not_ taking into account all those cases, just assuming that you have simple values between commas, you need something like that in props.conf stanza for your source/sourcetype/host:

SEDCMD-cutcolumn = s/^(([^,]*,){4})[^,]*,/\1/

 Might not work properly with less than 6 solumns present in the input

[gcusello]

Hi @super_saiyan,

are you speaking of exclusion from indexing (at index time) or from result displaying (at search time)?

Ciao.

Giuseppe

[super_saiyan]

Thanks for the quick response @gcusello 

yes, I'm referring to the exclusion from indexing.

could you please help

[gcusello]

Hi @super_saiyan,

as hinted by @PickleRick, you have to use the SEDCMD command or use props and transforms associated to the sourcetype you're using.

In few words, you have to find the regex to identify the column to exclude, e.g. if you have 100 columns divided by comma ",", you could use a regex like this:

in props.conf

[your_sourcetype]
TRANSFORMS-delete_column_80 = delete_column_80

in transforms.conf

[delete_column_80]
REGEX = ^(([^,]+,){80})[^,]+,(([^,]+,){19})
FORMAT = $1$2
DEST_KEY = _raw

For more infos see at https://docs.splunk.com/Documentation/Splunk/8.2.6/Data/Anonymizedata

Ciao.

Giuseppe