Help Using Props and/or Transforms to Mask sensitive field data at index time

[johnward4]

I have sensitive data that I'm attempting to mask at index time and I can't quite get the props and/or transforms to work. Please help

The sourcetype is : JMRequests

props.conf

[pw-mask]
SEDCMD-password = s/password:(\w|\d|\D)+)/password: XXXXXXXX/
TRANSFORMS = pw-mask

transforms.conf

[pw-mask]
REGEX = password(\w|\d|\D)+)
FORMAT = password:$1XXXXXXXX
DEST_KEY = _raw

Splunk is auto extracting the raw log data into fields successfully and the field that holds sensitive data is 'password'.

[mayurr98 · Accepted solution]

hey try this run anywhere search

| makeresults | eval raw="2018-01-24 02:08:26,114 [5756] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" username=\"jward\" password=\"pass$%$##@word9abc%#$$%@#$\" clientapplicationid=\"00000000-0000-0000-0000-000000000000\" parallelexecution=\"false\" languagecode=\"en\" xmlns=\"\"><Operation type=\"GetProjection\"><GetProjection><Incidents><Columns><Col name=\"c0\" mapping=\"UnRead\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c1\" mapping=\"MessageStat.HasMessageUnread\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c2\" mapping=\"ID\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c3\" mapping=\"TicketPriority.ForeColor\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c4\" mapping=\"KnownIssue\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c5\" mapping=\"Solicits\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c6\" mapping=\"TicketStat.ChildrenCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c7\" mapping=\"TicketStat.ConversationItemCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c8\" mapping=\"TicketStat.BlockedBy\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c9\" mapping=\"TicketStat.AttachmentCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c10\" mapping=\"Date\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" />" | rex field=raw mode=sed "s/password=\"([^\"]+)/password=\"XXXXXXX/g"

Just write this is in props.conf . you do not need to write transforms.conf.

[JMRequests]
SEDCMD-password = s/password=\"([^\"]+)/password=\"XXXXXXX/g

and then restart the server.
For more information, have a look at this doc
https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Anonymizedata#Replace_strings_with_regu...

If this does not work then try putting <your_source> instead of JMRequests.

let me know if this helps!

[johnward4]

The regex works in your first example but the props doesn't appear to work. My sourcetype for the log is sourcetype=JMRequests

[mayurr98]

1. Edit or create a copy of props.conf in $SPLUNK_HOME/etc/system/local on both indexer and forwarder

Create a props.conf stanza that uses SEDCMD to indicate a sed script:

[JMRequests]
 SEDCMD-password = s/password=\"([^\"]+)/password=\"XXXXXXX/g

2. Restart the server

Follow above carefully and do not skip any step. and write the props.conf in the path given above only

[johnward4]

Okay I think we're getting really close but the data is showing as \"XXXXXXX" now and it is not replacing null values as XXXXXXXX.

Also, the data has been indexing with clear text password values for a few weeks now, what is the best practice for masking the password values for already indexed data

[mayurr98]

show me the event of null password.

No you can't change data which is already index..you have reindex the file again.

[johnward4]

2018-01-27 08:26:26,119 [2640] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" username="jward" password="" clientapplicationid="00000000-0000-0000-0000-000000000000" parallelexecution="false" languagecode="en" xmlns=""><Operation type=""><LockObject><LockObjectRequest xmlns:xsd="http://www.w3.org/2001/XMLSchema"><ObjectID>T52383G</ObjectID><ObjectType>Ticket</ObjectType></LockObjectRequest></LockObject></Operation></Operations>

[mayurr98]

okay try this

 [JMRequests]
 SEDCMD-password = s/password=([^\s]+)/password="XXXXXXX"/g

this will be for both!.

accept my answer if this works for you.

[johnward4]

AWESOME! it works, thank you a ton

[mayurr98]

checklist:
1) have you restarted the server after configurtaion?
2) Are you doing this configuration on both forwarder and indexer?
If above checklist is "YES" for both question then
give me the artifacts of props..conf and tell me the path of props.conf

[johnward4]

unforunately, this is not an ideal solution to my issue.. these logs are being monitored and sent to my heavy forwarder and then my indexer. I'd like to use props and/or transforms on the heavy forwarder to mask the sensitive data.

[mayurr98]

yes i am telling you write that only this is just a cross-check if regex is working as expected or not! I have changed my answer pls check and follow the same

[mayurr98]

Can you provide sample event and tell what do you want to mask in that sample event?

[johnward4]

2018-01-24 02:08:26,114 [5756] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" username="kchee" password="password1" clientapplicationid="00000000-0000-0000-0000-000000000000" parallelexecution="false" languagecode="en" xmlns=""><Operation type="GetProjection"><GetProjection><Incidents><Columns><Col name="c0" mapping="UnRead" format="" culture="" maxlength="0" regex="" /><Col name="c1" mapping="MessageStat.HasMessageUnread" format="" culture="" maxlength="0" regex="" /><Col name="c2" mapping="ID" format="" culture="" maxlength="0" regex="" /><Col name="c3" mapping="TicketPriority.ForeColor" format="" culture="" maxlength="0" regex="" /><Col name="c4" mapping="KnownIssue" format="" culture="" maxlength="0" regex="" /><Col name="c5" mapping="Solicits" format="" culture="" maxlength="0" regex="" /><Col name="c6" mapping="TicketStat.ChildrenCount" format="" culture="" maxlength="0" regex="" /><Col name="c7" mapping="TicketStat.ConversationItemCount" format="" culture="" maxlength="0" regex="" /><Col name="c8" mapping="TicketStat.BlockedBy" format="" culture="" maxlength="0" regex="" /><Col name="c9" mapping="TicketStat.AttachmentCount" format="" culture="" maxlength="0" regex="" /><Col name="c10" mapping="Date" format="" culture="" maxlength="0" regex="" />