Remove fields at index time from a CSV file

[robwheeler]

Hello,

I have a CSV file that has 21 fields, I need to remove fields 1,2 & 15 from the CSV file as this contains sensitive information.

The list is comma separated but I can't work out how to remove these at index time. I have the option to remove or mask but the value in each field is of different length.

Can anyone help with the props & transforms required for this please.

Regards

Rob

[njandieri]

After several regex suicide commitments, I've got this below working:

SEDCMD-subst1 = s/(([^;]*;){NN})[^;]*;(.*)/\1SUBSTITUTION;\3/g

You can replace the semicolon with your delimiter, NN is an N-1 field to replace, SUBSTITUTION is the replacement text. You can have an empty string there to remove the field.

[richgalloway]

Try using SEDCMD. This is the method often used for masking, but can also omit fields. You may need to adjust the regex to match your data.

In props.conf:

[mySourcetype]
SEDCMD = s/([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)/\3,\4,\5,\6,\7,\8,\9,\10,\11,\12,\13,\14,\16,\17,\18,\19,\20,\21/