Getting Data In

Remove fields at index time from a CSV file

robwheeler
Engager

Hello,

I have a CSV file that has 21 fields, I need to remove fields 1,2 & 15 from the CSV file as this contains sensitive information.

The list is comma separated but I can't work out how to remove these at index time. I have the option to remove or mask but the value in each field is of different length.

Can anyone help with the props & transforms required for this please.

Regards

Rob

Tags (3)
0 Karma

njandieri
Explorer

After several regex suicide commitments, I've got this below working:

SEDCMD-subst1 = s/(([^;]*;){NN})[^;]*;(.*)/\1SUBSTITUTION;\3/g

You can replace the semicolon with your delimiter, NN is an N-1 field to replace, SUBSTITUTION is the replacement text. You can have an empty string there to remove the field.

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Try using SEDCMD. This is the method often used for masking, but can also omit fields. You may need to adjust the regex to match your data.

In props.conf:

[mySourcetype]
SEDCMD = s/([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*),([^,]*)/\3,\4,\5,\6,\7,\8,\9,\10,\11,\12,\13,\14,\16,\17,\18,\19,\20,\21/
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Enhance Your Splunk App Development: New Tools & Support

UCC FrameworkAdd-on Builder has been around for quite some time. It helps build Splunk apps faster, but it ...

Prove Your Splunk Prowess at .conf25—No Prereqs Required!

Your Next Big Security Credential: No Prerequisites Needed We know you’ve got the skills, and now, earning the ...

Splunk Observability Cloud's AI Assistant in Action Series: Observability as Code

This is the sixth post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...