topic Masking Sensitive Data in Splunk Search

Masking Sensitive Data

dperry — Fri, 30 Jan 2015 20:33:47 GMT

Ok Splunkers......

I have 1 search-head, 2 indexers, 1 Deployment server

Here is the event (sourcetype=mysourcetype)that I want to mask out the CC number:

2014-06-01 00:01:34 W3SVC1 10.0.99.120 GET /Disputes/BackToMYDomain.aspx ID=183481&ClaimNum=05/31-1370&DType=DMC&DClass=Debit%20Fraud&DeptExt=1234&**Card=1234567891234567**&SubmitDate=5/31/2014&samedayptr=N&CrdHolder=JOHN+DOE&TotClaim=150.00&Action=REVPC80MyDomain\mmouse10.XX.XX.XXMozilla/4.0+compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/5.0;+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E;+InfoPath.3) 200 0 0

I have the following props & Transforms in my Splunk_HOME/etc/system/local/directory:

props.conf:
[sourcetype]
TRANSFORMS-1card = cc_num_anon

Transforms.conf:
[cc_num_anon]
REGEX = (.*CARD=)\d{12}(\d{4}.*)
DEST_KEY = _raw
FORMAT = $1xxxxxxxxxxxx$2

I ran the debug mode, reloaded the forwarders....but Im still seeing the card information.....Oh I forgot to mention that the variable changes from Card/CARD within the event.

Re: Masking Sensitive Data

chanfoli — Fri, 30 Jan 2015 21:17:43 GMT

Normally, such masking will happen at parse time on the indexer, so just to be clear, these configs are on the indexers, correct?

A few things strike me as off about your regex:

REGEX = (.CARD=)d{12}(d{4}.)

I am not sure about the dots at the beginning and end, case is not matching your data either, also the character types don't have backslashes. So, to me this looks a little better:

REGEX = (Card=)\d{12}(\d{4})

Also note that this will only apply to newly indexed data once the transform is in place.

Re: Masking Sensitive Data

chanfoli — Fri, 30 Jan 2015 21:29:50 GMT

See updated answer.

Re: Masking Sensitive Data

martin_mueller — Fri, 30 Jan 2015 22:21:36 GMT

Try this props.conf-only solution that honours the variable case:

[sourcetype]
SEDCMD-cc = s/(?i)(card=)\d{12}(\d{4})/\1xxxxxxxxxxxx\2/g

This should also run considerably faster because a leading .* in a regex will eat your server's soul.

Re: Masking Sensitive Data

dperry — Sat, 31 Jan 2015 03:24:40 GMT

Silly question...when you say add to the props.conf...this is on the indexers, correct?

Indexer
Splunk_HOME/etc/system/local

The deployment server in Deployment apps (the app) local folder.

Re: Masking Sensitive Data

dperry — Sat, 31 Jan 2015 04:41:44 GMT

So Im having a different scenario since I added the SEDCMD to the indexer local props.conf:

Right output: (IT WORKS)

2015-01-31 03:58:10 W3SVC1 10.XX.XX.XX GET /Disputes/BackTohost.aspx ID=222888&ShID=70&
Choice=Fraud&CARD=xxxxxxxxxxxx5144&DType=DEBIT&ACode=W&SCode=G&AmtFr=&AmtTo=&DtFr=01/29/2015&DtTo=01/30/2015&CCIssued=&
Action=History 80 MyDomain\mmouse 10.XX.XX.XX Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/5.0;+SLCC2;
+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E;+InfoPath.3) 200 0 0

Then I get a different output from the same sourcetype: (Its not changing the Card=, but adding an end output)

2015-01-30 20:33:52 W3SVC1 10.XX.XX.XX GET /Disputes/BackTohost.aspx ID=222796&
ShID=ALL%20SHARES&Choice=Fraud&CARD=1234567891234567&DType=DEBIT&ACode=W&SCode=G&AmtFr=&AmtTo=&DtFr=&DtTo=&
CCIssued=&Action=History 80 MyDoamin\mmouse 10.XX.XX.XX Mozilla/4.0+(compatible;+MSIE+7.0;+Windows+NT+6.1;+WOW64;+Trident/5.0;
+SLCC2;+.NET+CLR+2.0.50727;+.NET+CLR+3.5.30729;+.NET+CLR+3.0.30729;+Media+Center+PC+6.0;+.NET4.0C;+.NET4.0E) 200 0 0xxxxxxxxxxxx$2

The the second event it applying the numbers 0xxxxxxxxxxxx$2

Is this because the event is different in word count?

Re: Masking Sensitive Data

martin_mueller — Sat, 31 Jan 2015 12:45:20 GMT

This config belongs on the indexers. If you have heavy forwarders that perform parsing then you will need it there as well.

The second event is several hours older, it was indexed before the SEDCMD was applied. I'm sure of this because there is $2 at the end, and there is no dollar sign used in the SEDCMD.