topic Re: Data Masking Before Ingestion in Security

Data Masking Before Ingestion

anandhalagaras1 — Thu, 21 Mar 2024 06:14:27 GMT

Hi Team,

Want to mask two of the fields "password" and "cpassword" from the events which are getting written with the plain text information. So needs to be changed as #####.

Sample event information:

[2024-01-31_07:58:28] INFO : REQUEST: User:abc CreateUser POST: name: AB_Test_Max;email: xyz@gmail.com;password: abc12345679;cpassword: abc12345679;role: User;

[2024-01-30_14:05:42] INFO : REQUEST: User:xyz CreateUser POST: name: Math_Lab;email: abc@yahoo.com;password: xyzab54;cpassword: xyzab54;role: Admin;

So kindly help with the props.conf so that i can apply with SEDCMD-mask.

Re: Data Masking Before Ingestion

gcusello — Thu, 21 Mar 2024 06:25:58 GMT

Hi @anandhalagaras1,please try this:

[your_sourcetype] SEDCMD = s/password: ([^;]+);cpassword: ([^;]+);/password: (####);cpassword: (####);/gm

that you can test at https://regex101.com/r/ppaFZc/1

Ciao.

Giuseppe

Re: Data Masking Before Ingestion

anandhalagaras1 — Mon, 01 Apr 2024 11:11:49 GMT

@gcusello

We had two requirements for the same sourcetype. One involved line breaks, and the other required password masking during ingestion. As our Search heads are managed by Splunk Support and hosted in the Cloud, we created a custom app and deployed the props.conf in the default directory. After uploading the apps for the cloud vetting process, they were successfully installed. However, I've noticed that the logs are now being separated into individual events, which is acceptable, but the passwords are still visible and haven't been masked according to our requirement. I'm unsure about where exactly I may have missed it.

This is the props.conf file for reference.

[sourcetype]
SHOULD_LINEMERGE = false
SEDCMD = s/password: ([^;]+);cpassword: ([^;]+);/password: (####);cpassword: (####);/gm

Sample log for reference:

[2024-03-01_06:32:08] INFO : REQUEST: User:abc CreateUser POST: name: xyz;email: abc@gmail.com;password: xyz@123;cpassword: xyz@123;role: Administrator;

So kindly help on this requirement.

Re: Data Masking Before Ingestion

gcusello — Tue, 02 Apr 2024 07:25:03 GMT

Hi @anandhalagaras1,

regex substitution is correct.

Are you sure about the sourcetype?

is there any sourcetype replacement in your data?

are there some other Heavy Forwarders before the one you used for the masking?

Ciao.

Giuseppe

Re: Data Masking Before Ingestion

anandhalagaras1 — Wed, 03 Apr 2024 11:55:46 GMT

@gcusello ,

This is the exact and correct sourcetype and I have created a custom app and uploaded the App in our Search head. Since our Search head is hosted in Splunk Cloud managed by Support.

So I have uploaded the app in the upload app section and post vetting process completed i have installed the custom app into the Search head.

This is the custom app i have created "abc_app"

Under abc_app I have placed two folders "default" and "metadata"

Under default I have created the app.conf and props.conf

And under metadata I have created the default.metadata

Refer screenshots for reference.

So kindly let me know where i am missing since the lines are getting segregated as separate events whereas password masking is not getting applied to the events. Hence kindly help on the same.

Re: Data Masking Before Ingestion

gcusello — Wed, 03 Apr 2024 12:24:12 GMT

Hi @anandhalagaras1,

what's the sourcetype to apply the masking?

I suppose that sourcetype in the props.conf stanza header it's only for example and that in your installation you have the correct sourcetype to apply the transformation.

ciao.

Giuseppe

Re: Data Masking Before Ingestion

anandhalagaras1 — Wed, 03 Apr 2024 15:10:26 GMT

@gcusello Indeed, I have applied the correct sourcetype there to ensure that events are appropriately divided. Nonetheless, the masking of passwords is not taking place as intended.

Re: Data Masking Before Ingestion

anandhalagaras1 — Mon, 15 Apr 2024 10:41:19 GMT

@gcusello ,

Any inputs from your end since still i can see the events are getting ingested with the password information present in it.

Re: Data Masking Before Ingestion

marnall — Mon, 15 Apr 2024 21:00:25 GMT

Could you try this SEDCMD in the props.conf file? (Make sure that the stanza is changed to match the sourcetype of the logs)

[your_sourcetype] SEDCMD-maskpasswords = s/password: ([^;]+);cpassword: ([^;]+);/password: ####;cpassword: ####;/g