topic Remove fields at index time from a CSV file in Getting Data In

Remove fields at index time from a CSV file

robwheeler — Fri, 18 Aug 2017 16:12:19 GMT

Hello,

I have a CSV file that has 21 fields, I need to remove fields 1,2 & 15 from the CSV file as this contains sensitive information.

The list is comma separated but I can't work out how to remove these at index time. I have the option to remove or mask but the value in each field is of different length.

Can anyone help with the props & transforms required for this please.

Regards

Rob

Re: Remove fields at index time from a CSV file

richgalloway — Fri, 18 Aug 2017 17:22:57 GMT

Try using SEDCMD. This is the method often used for masking, but can also omit fields. You may need to adjust the regex to match your data.

In props.conf:

[mySourcetype]
SEDCMD = s/([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)/\3,\4,\5,\6,\7,\8,\9,\10,\11,\12,\13,\14,\16,\17,\18,\19,\20,\21/

Re: Remove fields at index time from a CSV file

njandieri — Sat, 09 Mar 2019 09:40:03 GMT

After several regex suicide commitments, I've got this below working:

SEDCMD-subst1 = s/(([^;]*;){NN})[^;]*;(.*)/\1SUBSTITUTION;\3/g

You can replace the semicolon with your delimiter, NN is an N-1 field to replace, SUBSTITUTION is the replacement text. You can have an empty string there to remove the field.