Getting Data In

Help Using Props and/or Transforms to Mask sensitive field data at index time

johnward4
Communicator

I have sensitive data that I'm attempting to mask at index time and I can't quite get the props and/or transforms to work. Please help

The sourcetype is : JMRequests

props.conf

[pw-mask]
SEDCMD-password = s/password:(\w|\d|\D)+)/password: XXXXXXXX/
TRANSFORMS = pw-mask

transforms.conf

[pw-mask]
REGEX = password(\w|\d|\D)+)
FORMAT = password:$1XXXXXXXX
DEST_KEY = _raw

Splunk is auto extracting the raw log data into fields successfully and the field that holds sensitive data is 'password'.

0 Karma
1 Solution

mayurr98
Super Champion

hey try this run anywhere search

| makeresults | eval raw="2018-01-24 02:08:26,114 [5756] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" username=\"jward\" password=\"pass$%$##@word9abc%#$$%@#$\" clientapplicationid=\"00000000-0000-0000-0000-000000000000\" parallelexecution=\"false\" languagecode=\"en\" xmlns=\"\"><Operation type=\"GetProjection\"><GetProjection><Incidents><Columns><Col name=\"c0\" mapping=\"UnRead\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c1\" mapping=\"MessageStat.HasMessageUnread\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c2\" mapping=\"ID\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c3\" mapping=\"TicketPriority.ForeColor\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c4\" mapping=\"KnownIssue\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c5\" mapping=\"Solicits\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c6\" mapping=\"TicketStat.ChildrenCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c7\" mapping=\"TicketStat.ConversationItemCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c8\" mapping=\"TicketStat.BlockedBy\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c9\" mapping=\"TicketStat.AttachmentCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c10\" mapping=\"Date\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" />" | rex field=raw mode=sed "s/password=\"([^\"]+)/password=\"XXXXXXX/g"

Just write this is in props.conf . you do not need to write transforms.conf.

[JMRequests]
SEDCMD-password = s/password=\"([^\"]+)/password=\"XXXXXXX/g

and then restart the server.
For more information, have a look at this doc
https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Anonymizedata#Replace_strings_with_regu...

If this does not work then try putting <your_source> instead of JMRequests.

let me know if this helps!

View solution in original post

0 Karma

mayurr98
Super Champion

hey try this run anywhere search

| makeresults | eval raw="2018-01-24 02:08:26,114 [5756] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" username=\"jward\" password=\"pass$%$##@word9abc%#$$%@#$\" clientapplicationid=\"00000000-0000-0000-0000-000000000000\" parallelexecution=\"false\" languagecode=\"en\" xmlns=\"\"><Operation type=\"GetProjection\"><GetProjection><Incidents><Columns><Col name=\"c0\" mapping=\"UnRead\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c1\" mapping=\"MessageStat.HasMessageUnread\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c2\" mapping=\"ID\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c3\" mapping=\"TicketPriority.ForeColor\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c4\" mapping=\"KnownIssue\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c5\" mapping=\"Solicits\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c6\" mapping=\"TicketStat.ChildrenCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c7\" mapping=\"TicketStat.ConversationItemCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c8\" mapping=\"TicketStat.BlockedBy\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c9\" mapping=\"TicketStat.AttachmentCount\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" /><Col name=\"c10\" mapping=\"Date\" format=\"\" culture=\"\" maxlength=\"0\" regex=\"\" />" | rex field=raw mode=sed "s/password=\"([^\"]+)/password=\"XXXXXXX/g"

Just write this is in props.conf . you do not need to write transforms.conf.

[JMRequests]
SEDCMD-password = s/password=\"([^\"]+)/password=\"XXXXXXX/g

and then restart the server.
For more information, have a look at this doc
https://docs.splunk.com/Documentation/SplunkCloud/6.6.3/Data/Anonymizedata#Replace_strings_with_regu...

If this does not work then try putting <your_source> instead of JMRequests.

let me know if this helps!

0 Karma

johnward4
Communicator

The regex works in your first example but the props doesn't appear to work. My sourcetype for the log is sourcetype=JMRequests

0 Karma

mayurr98
Super Champion
  1. Edit or create a copy of props.conf in $SPLUNK_HOME/etc/system/local on both indexer and forwarder

Create a props.conf stanza that uses SEDCMD to indicate a sed script:

[JMRequests]
 SEDCMD-password = s/password=\"([^\"]+)/password=\"XXXXXXX/g

2. Restart the server

Follow above carefully and do not skip any step. and write the props.conf in the path given above only

0 Karma

johnward4
Communicator

Okay I think we're getting really close but the data is showing as \"XXXXXXX" now and it is not replacing null values as XXXXXXXX.

Also, the data has been indexing with clear text password values for a few weeks now, what is the best practice for masking the password values for already indexed data

0 Karma

mayurr98
Super Champion

show me the event of null password.

No you can't change data which is already index..you have reindex the file again.

0 Karma

johnward4
Communicator
2018-01-27 08:26:26,119 [2640] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" username="jward" password="" clientapplicationid="00000000-0000-0000-0000-000000000000" parallelexecution="false" languagecode="en" xmlns=""><Operation type=""><LockObject><LockObjectRequest xmlns:xsd="http://www.w3.org/2001/XMLSchema"><ObjectID>T52383G</ObjectID><ObjectType>Ticket</ObjectType></LockObjectRequest></LockObject></Operation></Operations>
0 Karma

mayurr98
Super Champion

okay try this

 [JMRequests]
 SEDCMD-password = s/password=([^\s]+)/password="XXXXXXX"/g

this will be for both!.

accept my answer if this works for you.

0 Karma

johnward4
Communicator

AWESOME! it works, thank you a ton

0 Karma

mayurr98
Super Champion

checklist:
1) have you restarted the server after configurtaion?
2) Are you doing this configuration on both forwarder and indexer?
If above checklist is "YES" for both question then
give me the artifacts of props..conf and tell me the path of props.conf

0 Karma

johnward4
Communicator

unforunately, this is not an ideal solution to my issue.. these logs are being monitored and sent to my heavy forwarder and then my indexer. I'd like to use props and/or transforms on the heavy forwarder to mask the sensitive data.

0 Karma

mayurr98
Super Champion

yes i am telling you write that only this is just a cross-check if regex is working as expected or not! I have changed my answer pls check and follow the same

0 Karma

mayurr98
Super Champion

Can you provide sample event and tell what do you want to mask in that sample event?

0 Karma

johnward4
Communicator
2018-01-24 02:08:26,114 [5756] INFO  - REQUEST (ExecuteEx, xml inline): <Operations xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" username="kchee" password="password1" clientapplicationid="00000000-0000-0000-0000-000000000000" parallelexecution="false" languagecode="en" xmlns=""><Operation type="GetProjection"><GetProjection><Incidents><Columns><Col name="c0" mapping="UnRead" format="" culture="" maxlength="0" regex="" /><Col name="c1" mapping="MessageStat.HasMessageUnread" format="" culture="" maxlength="0" regex="" /><Col name="c2" mapping="ID" format="" culture="" maxlength="0" regex="" /><Col name="c3" mapping="TicketPriority.ForeColor" format="" culture="" maxlength="0" regex="" /><Col name="c4" mapping="KnownIssue" format="" culture="" maxlength="0" regex="" /><Col name="c5" mapping="Solicits" format="" culture="" maxlength="0" regex="" /><Col name="c6" mapping="TicketStat.ChildrenCount" format="" culture="" maxlength="0" regex="" /><Col name="c7" mapping="TicketStat.ConversationItemCount" format="" culture="" maxlength="0" regex="" /><Col name="c8" mapping="TicketStat.BlockedBy" format="" culture="" maxlength="0" regex="" /><Col name="c9" mapping="TicketStat.AttachmentCount" format="" culture="" maxlength="0" regex="" /><Col name="c10" mapping="Date" format="" culture="" maxlength="0" regex="" />
0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...