topic Re: Help Using Props and/or Transforms to Mask sensitive field data at index time in Getting Data In

Help Using Props and/or Transforms to Mask sensitive field data at index time

johnward4 — Sat, 27 Jan 2018 02:38:36 GMT

I have sensitive data that I'm attempting to mask at index time and I can't quite get the props and/or transforms to work. Please help

The sourcetype is : JMRequests

props.conf

[pw-mask]
SEDCMD-password = s/password:(\w|\d|\D)+)/password: XXXXXXXX/
TRANSFORMS = pw-mask

transforms.conf

[pw-mask]
REGEX = password(\w|\d|\D)+)
FORMAT = password:$1XXXXXXXX
DEST_KEY = _raw

Splunk is auto extracting the raw log data into fields successfully and the field that holds sensitive data is 'password'.

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

mayurr98 — Sat, 27 Jan 2018 03:03:31 GMT

Can you provide sample event and tell what do you want to mask in that sample event?

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

johnward4 — Sat, 27 Jan 2018 03:11:29 GMT

2018-01-24 02:08:26,114 [5756] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" username="kchee" password="password1" clientapplicationid="00000000-0000-0000-0000-000000000000" parallelexecution="false" languagecode="en" xmlns=""><Operation type="GetProjection"><GetProjection><Incidents><Columns><Col name="c0" mapping="UnRead" format="" culture="" maxlength="0" regex="" /><Col name="c1" mapping="MessageStat.HasMessageUnread" format="" culture="" maxlength="0" regex="" /><Col name="c2" mapping="ID" format="" culture="" maxlength="0" regex="" /><Col name="c3" mapping="TicketPriority.ForeColor" format="" culture="" maxlength="0" regex="" /><Col name="c4" mapping="KnownIssue" format="" culture="" maxlength="0" regex="" /><Col name="c5" mapping="Solicits" format="" culture="" maxlength="0" regex="" /><Col name="c6" mapping="TicketStat.ChildrenCount" format="" culture="" maxlength="0" regex="" /><Col name="c7" mapping="TicketStat.ConversationItemCount" format="" culture="" maxlength="0" regex="" /><Col name="c8" mapping="TicketStat.BlockedBy" format="" culture="" maxlength="0" regex="" /><Col name="c9" mapping="TicketStat.AttachmentCount" format="" culture="" maxlength="0" regex="" /><Col name="c10" mapping="Date" format="" culture="" maxlength="0" regex="" />

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

mayurr98 — Sat, 27 Jan 2018 03:24:08 GMT

hey try this run anywhere search

| makeresults | eval raw="2018-01-24 02:08:26,114 [5756] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" username=\"jward\" password=\"pass$%$##@word9abc%#$$%@#$\" clientapplicationid=\"00000000-0000-0000-0000-000000000000\" parallelexecution=\"false\" languagecode=\"en\" xmlns=\"\"><Operation type=\"GetProjection\"><GetProjection><Incidents><Columns><Col name=\"c0\" mapping=\"UnRead\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c1\" mapping=\"MessageStat.HasMessageUnread\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c2\" mapping=\"ID\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c3\" mapping=\"TicketPriority.ForeColor\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c4\" mapping=\"KnownIssue\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c5\" mapping=\"Solicits\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c6\" mapping=\"TicketStat.ChildrenCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c7\" mapping=\"TicketStat.ConversationItemCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c8\" mapping=\"TicketStat.BlockedBy\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c9\" mapping=\"TicketStat.AttachmentCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c10\" mapping=\"Date\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" />" | rex field=raw mode=sed "s/password=\"([^\"]+)/password=\"XXXXXXX/g"

Just write this is in props.conf . you do not need to write transforms.conf.

[JMRequests]
SEDCMD-password = s/password=\"([^\"]+)/password=\"XXXXXXX/g

and then restart the server.
For more information, have a look at this doc
https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Anonymizedata#Replace_strings_with_regular_expression_match

If this does not work then try putting <your_source> instead of JMRequests.

let me know if this helps!

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

johnward4 — Sat, 27 Jan 2018 03:28:13 GMT

unforunately, this is not an ideal solution to my issue.. these logs are being monitored and sent to my heavy forwarder and then my indexer. I'd like to use props and/or transforms on the heavy forwarder to mask the sensitive data.

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

mayurr98 — Sat, 27 Jan 2018 03:32:18 GMT

yes i am telling you write that only this is just a cross-check if regex is working as expected or not! I have changed my answer pls check and follow the same

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

johnward4 — Sat, 27 Jan 2018 04:38:59 GMT

The regex works in your first example but the props doesn't appear to work. My sourcetype for the log is sourcetype=JMRequests

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

mayurr98 — Sat, 27 Jan 2018 04:56:06 GMT

checklist:
1) have you restarted the server after configurtaion?
2) Are you doing this configuration on both forwarder and indexer?
If above checklist is "YES" for both question then
give me the artifacts of props..conf and tell me the path of props.conf

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

mayurr98 — Sat, 27 Jan 2018 05:13:41 GMT

Edit or create a copy of props.conf in $SPLUNK_HOME/etc/system/local on both indexer and forwarder

Create a props.conf stanza that uses SEDCMD to indicate a sed script:

[JMRequests]
 SEDCMD-password = s/password=\"([^\"]+)/password=\"XXXXXXX/g

2. Restart the server

Follow above carefully and do not skip any step. and write the props.conf in the path given above only

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

johnward4 — Sat, 27 Jan 2018 06:58:33 GMT

Okay I think we're getting really close but the data is showing as \"XXXXXXX" now and it is not replacing null values as XXXXXXXX.

Also, the data has been indexing with clear text password values for a few weeks now, what is the best practice for masking the password values for already indexed data

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

mayurr98 — Sat, 27 Jan 2018 07:06:01 GMT

show me the event of null password.

No you can't change data which is already index..you have reindex the file again.

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

johnward4 — Sat, 27 Jan 2018 07:28:09 GMT

2018-01-27 08:26:26,119 [2640] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" username="jward" password="" clientapplicationid="00000000-0000-0000-0000-000000000000" parallelexecution="false" languagecode="en" xmlns=""><Operation type=""><LockObject><LockObjectRequest xmlns:xsd="http://www.w3.org/2001/XMLSchema"><ObjectID>T52383G</ObjectID><ObjectType>Ticket</ObjectType></LockObjectRequest></LockObject></Operation></Operations>

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

mayurr98 — Sat, 27 Jan 2018 07:47:51 GMT

okay try this

 [JMRequests]
 SEDCMD-password = s/password=([^\s]+)/password="XXXXXXX"/g

this will be for both!.

accept my answer if this works for you.

Re: Help Using Props and/or Transforms to Mask sensitive field data at index time

johnward4 — Sat, 27 Jan 2018 08:25:40 GMT

AWESOME! it works, thank you a ton