Getting Data In
Highlighted

Help Using Props and/or Transforms to Mask sensitive field data at index time

Communicator

I have sensitive data that I'm attempting to mask at index time and I can't quite get the props and/or transforms to work. Please help

The sourcetype is : JMRequests

props.conf

[pw-mask]
SEDCMD-password = s/password:(\w|\d|\D)+)/password: XXXXXXXX/
TRANSFORMS = pw-mask

transforms.conf

[pw-mask]
REGEX = password(\w|\d|\D)+)
FORMAT = password:$1XXXXXXXX
DEST_KEY = _raw

Splunk is auto extracting the raw log data into fields successfully and the field that holds sensitive data is 'password'.

0 Karma
Highlighted

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

SplunkTrust
SplunkTrust

Can you provide sample event and tell what do you want to mask in that sample event?

0 Karma
Highlighted

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

Communicator
2018-01-24 02:08:26,114 [5756] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" username="kchee" password="password1" clientapplicationid="00000000-0000-0000-0000-000000000000" parallelexecution="false" languagecode="en" xmlns=""><Operation type="GetProjection"><GetProjection><Incidents><Columns><Col name="c0" mapping="UnRead" format="" culture="" maxlength="0" regex="" /><Col name="c1" mapping="MessageStat.HasMessageUnread" format="" culture="" maxlength="0" regex="" /><Col name="c2" mapping="ID" format="" culture="" maxlength="0" regex="" /><Col name="c3" mapping="TicketPriority.ForeColor" format="" culture="" maxlength="0" regex="" /><Col name="c4" mapping="KnownIssue" format="" culture="" maxlength="0" regex="" /><Col name="c5" mapping="Solicits" format="" culture="" maxlength="0" regex="" /><Col name="c6" mapping="TicketStat.ChildrenCount" format="" culture="" maxlength="0" regex="" /><Col name="c7" mapping="TicketStat.ConversationItemCount" format="" culture="" maxlength="0" regex="" /><Col name="c8" mapping="TicketStat.BlockedBy" format="" culture="" maxlength="0" regex="" /><Col name="c9" mapping="TicketStat.AttachmentCount" format="" culture="" maxlength="0" regex="" /><Col name="c10" mapping="Date" format="" culture="" maxlength="0" regex="" />
0 Karma
Highlighted

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

SplunkTrust
SplunkTrust

hey try this run anywhere search

| makeresults | eval raw="2018-01-24 02:08:26,114 [5756] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" username=\"jward\" password=\"pass$%$##@word9abc%#$$%@#$\" clientapplicationid=\"00000000-0000-0000-0000-000000000000\" parallelexecution=\"false\" languagecode=\"en\" xmlns=\"\"><Operation type=\"GetProjection\"><GetProjection><Incidents><Columns><Col name=\"c0\" mapping=\"UnRead\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c1\" mapping=\"MessageStat.HasMessageUnread\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c2\" mapping=\"ID\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c3\" mapping=\"TicketPriority.ForeColor\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c4\" mapping=\"KnownIssue\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c5\" mapping=\"Solicits\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c6\" mapping=\"TicketStat.ChildrenCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c7\" mapping=\"TicketStat.ConversationItemCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c8\" mapping=\"TicketStat.BlockedBy\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c9\" mapping=\"TicketStat.AttachmentCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c10\" mapping=\"Date\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" />" | rex field=raw mode=sed "s/password=\"([^\"]+)/password=\"XXXXXXX/g"

Just write this is in props.conf . you do not need to write transforms.conf.

[JMRequests]
SEDCMD-password = s/password=\"([^\"]+)/password=\"XXXXXXX/g

and then restart the server.
For more information, have a look at this doc
https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Anonymizedata#Replace_strings_with_regu...

If this does not work then try putting <your_source> instead of JMRequests.

let me know if this helps!

View solution in original post

0 Karma
Highlighted

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

Communicator

unforunately, this is not an ideal solution to my issue.. these logs are being monitored and sent to my heavy forwarder and then my indexer. I'd like to use props and/or transforms on the heavy forwarder to mask the sensitive data.

0 Karma
Highlighted

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

SplunkTrust
SplunkTrust

yes i am telling you write that only this is just a cross-check if regex is working as expected or not! I have changed my answer pls check and follow the same

0 Karma
Highlighted

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

Communicator

The regex works in your first example but the props doesn't appear to work. My sourcetype for the log is sourcetype=JMRequests

0 Karma
Highlighted

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

SplunkTrust
SplunkTrust

checklist:
1) have you restarted the server after configurtaion?
2) Are you doing this configuration on both forwarder and indexer?
If above checklist is "YES" for both question then
give me the artifacts of props..conf and tell me the path of props.conf

0 Karma
Highlighted

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

SplunkTrust
SplunkTrust
  1. Edit or create a copy of props.conf in $SPLUNK_HOME/etc/system/local on both indexer and forwarder

Create a props.conf stanza that uses SEDCMD to indicate a sed script:

[JMRequests]
 SEDCMD-password = s/password=\"([^\"]+)/password=\"XXXXXXX/g

2. Restart the server

Follow above carefully and do not skip any step. and write the props.conf in the path given above only

0 Karma
Highlighted

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

Communicator

Okay I think we're getting really close but the data is showing as \"XXXXXXX" now and it is not replacing null values as XXXXXXXX.

Also, the data has been indexing with clear text password values for a few weeks now, what is the best practice for masking the password values for already indexed data

0 Karma