topic Re: Not able to mask JSON before/at the time of indexing in Getting Data In

Not able to mask JSON before/at the time of indexing

meenal901 — Tue, 29 Sep 2020 19:43:28 GMT

Hi,

I have json data to be parsed and there is a field "password" which needs to be masked at index time. Following the wiki page https://wiki.splunk.com/Community:How_to_mask_strings_in_json_event_at_Indexing_time_when_using_INDEXED_EXTRACTION I am able to mask the _raw but the field still shows plain text.

Here's is my props and transforms and screenshot from search head:

** props.conf **
test_json_passwordINDEXED_EXTRACTIONS = json
TIMESTAMP_FIELDS = @timestamp
SEDCMD-mask_password_raw = s/\S+( - password)/"######\1/
TRANSFORMS-mask_json_password = mask_json_password_meta
KV_MODE = none
category = Structured
description = JavaScript Object Notation format. For more information, visit http://json.org/
disabled = false
pulldown_type = true

** transforms.conf **
[mask_json_password_meta]
SOURCE_KEY = _meta
DEST_KEY = _meta
REGEX = ^(.message[tT]ext::)\S+ - password" (.)
FORMAT = $1"###### - password" $2
WRITE_META = false

** Sample Data **
{
"username" : "my_username",
"password" : "my_password - password",
"validation-factors" : {
"validationFactors" : [
{
"name" : "remote_address",
"value" : "127.0.0.1"
}
]
},
"@timestamp": "2018-01-05T14:56:29.000Z",
"attributes": {
"field_1": "value_1",
"field_2": "value_2",
"field_3": "value_3"
}
}
1) Masked data in raw event

2) Clear password visible in password field

Re: Not able to mask JSON before/at the time of indexing

cpetterborg — Wed, 30 May 2018 11:30:51 GMT

You can't do it on the search head. Your props.conf and transforms.conf must be on the indexers for this to work. It should be possible to do this with only the props file.

Re: Not able to mask JSON before/at the time of indexing

FrankVl — Tue, 29 Sep 2020 19:43:36 GMT

Given that the SEDCMD works, the config must be on his indexers already.

Could it be that the "indexed extractions" take place before the SEDCMD? Solution may be to define a specific TRANSFORMS with the password field as SOURCE_KEY and DEST_KEY and then apply the masking again.

Edit: Oh, and shoot the person who thought it was a good idea to write plaintext passwords to a log in the first place 😛

Re: Not able to mask JSON before/at the time of indexing

meenal901 — Tue, 29 Sep 2020 19:43:41 GMT

Oh I wish I could 😛

I will try the SOURCE_KEY and DEST_KEY as password and try again.