Unable to anonymise data even after we have right regex.

[vikram_m]

I have a log file as detailed below.

"02d4224f-8cc8-48b4-a06d-034a97ad68a8-333999","2018-03-24T04:58:09.280Z","OPENIG-HTTP-ACCESS","02d4224f-8cc8-48b4-a06d-034a97ad68a8-333998",,,"10.255.0.19","8443","10.255.0.11","11700",,,,"true","GET","https://imsdat.sentienceanalytics.com:443/sentience/iams/api/auth/services/system","{}","{""activity... eyAidHlwIjogIkpXVCIsICJraWQiOiAiaFNsNTU4b2NrZHpsWmFHTXpweUZUOGZSamg0PSIsICJhbGciOiAiUlMyNTYiIH0.eyAiYXRfaGFzaCI6ICJIemJPcTJkLTB5LThIWThMOS1ZdFVnIiwgInN1YiI6ICI5MDFkMTMwNy00NjdkLTRmZDctOTEzMC1lNzUwM2U5OGFhOWEiLCAiYXVkaXRUcmFja2luZ0lkIjogIjkyMDk5NzcyLWU0MWEtNGQwYi1hYmM1LTg2NzY3MGIzYTIyNC01NTg2NDQiLCAiaXNzIjogImh0dHBzOi8vaW1zZGF0LnNlbnRpZW5jZWFuYWx5dGljcy5jb206NDQzL29wZW5hbS9vYXV0aDIvY2JwZ2ZiZGV2IiwgInRva2VuTmFtZSI6ICJpZF90b2tlbiIsICJhdWQiOiAiOTAxZDEzMDctNDY3ZC00ZmQ3LTkxMzAtZTc1MDNlOThhYTlhIiwgIm9yZy5mb3JnZXJvY2sub3BlbmlkY29ubmVjdC5vcHMiOiAiZDg5NTM5ZTUtNDA3Mi00MDNkLWExYjEtM2Q2M2FkZmI5ZmVjIiwgImF6cCI6ICI5MDFkMTMwNy00NjdkLTRmZDctOTEzMC1lNzUwM2U5OGFhOWEiLCAiYXV0aF90aW1lIjogMTUyMTg2NzA2MSwgInJlYWxtIjogIi9jYnBnZmJkZXYiLCAiZXhwIjogMTUyMTg3MDY2MSwgInRva2VuVHlwZSI6ICJKV1RUb2tlbiIsICJpYXQiOiAxNTIxODY3MDYxIH0.WO7PoFhmieqV0tK03Zft0gUY2fDzd6-h0FJQtCoAWtUbiK_5v2OY-wbpXyMK3GC_XlOX4VfbavjBE4K00ryzhZsaQ2CZA-uQGgeDELQEsxk_yNQyex00WpPpq0Xg8S_4gNBDg3HvfrlDrGD6MYoWRhrP2K3i6ggbSqdCPu5YSqIRF9P06WqeoD01z4tm4D_4tG-Z17_EuXllUAmtoPqN0ZKrYda56HsS-u6EuRqHvUViHIeawCwTgzdkvGMzvWs-SoP5-_djHUP7mZbI2I4CoS_eDTEbTiJ8uYOyvhXnLEqmOjy4FYIHHqP6E4yA2bGDfVCtj85ixOEZfxLCp_yG9g""],""host"":[""imsdat.sentienceanalytics.com""],""user-agent"":[""Honeywell.Sentience.WebApiClient.Identity.Rest.SentienceIdentityPublicWebApis/1.0.0-ci.182+Branch.master.Sha.0a4c3e515876f9ce616d0bf07bd1fe9bb7831f6c""]}","{}","{""Cache-Control"":[""no-cache, no-store, max-age=0, must-revalidate""],""Content-Type"":[""application/json; charset=UTF-8""],""Date"":[""Sat, 24 Mar 2018 04:58:09 GMT""],""Expires"":[""0""],""Pragma"":[""no-cache""],""RequestId"":[""02d4224f-8cc8-48b4-a06d-034a97ad68a8-333998""],""X-Content-Type-Options"":[""nosniff""],""X-Frame-Options"":[""DENY""],""X-XSS-Protection"":[""1; mode=block""]}","SUCCESSFUL","200",,"18","MILLISECONDS"

When I wrote the SED command in Linux and checked I am able to see the Bearer field is hashed.

[splunk@splunkqadeployer ~]$ sed 's/Bearer (.{1150})/#####/g' /tmp/test.anon
"02d4224f-8cc8-48b4-a06d-034a97ad68a8-334002","2018-03-24T04:58:09.313Z","OPENIG-HTTP-ACCESS","02d4224f-8cc8-48b4-a06d-034a97ad68a8-334001",,,"10.255.0.19","8443","10.255.0.11","11700",,,,"true","POST","https://imsdat.sentienceanalytics.com:443/sentience/iams/api/auth/systems/8a5bc996-8045-41af-b137-71...; charset=utf-8""],""host"":[""imsdat.sentienceanalytics.com""],""user-agent"":[""Honeywell.Sentience.WebApiClient.Identity.Rest.SentienceIdentityPublicWebApis/1.0.0-ci.182+Branch.master.Sha.0a4c3e515876f9ce616d0bf07bd1fe9bb7831f6c""]}","{}","{""Cache-Control"":[""no-cache, no-store, max-age=0, must-revalidate""],""Content-Length"":[""0""],""Date"":[""Sat, 24 Mar 2018 04:58:09 GMT""],""Expires"":[""0""],""Pragma"":[""no-cache""],""RequestId"":[""02d4224f-8cc8-48b4-a06d-034a97ad68a8-334001""],""X-Content-Type-Options"":[""nosniff""],""X-Frame-Options"":[""DENY""],""X-XSS-Protection"":[""1; mode=block""]}","SUCCESSFUL","200",,"75","MILLISECONDS"

I had put this thing into props.conf with below parameters in place.

[splunk@splunkqadeployer local]$ cat props.conf
[SOURCETYPE]
SEDCMD-Bearer = s/Bearer (.{1150})/#####/g

I deployed this app into the indexer and UF from where these files are comming.

after multiple efforts I am unable to see the data masked. May I know where I am missing here. As RegEx seems fine by the app deployment is getting failed for me.

[mayurr98]

you need to put this in indexer or Heavy forwarder mostly in splunk/etc/apps/<app-name>/local.
If you have indexer-cluster then put this configuration in master-apps/<appname>/props.conf on the cluster master and then push the bundle from the cluster master GUI.
This configuration will apply to latest & future events and it will not apply to historical events. So check latest events only.
let me know if this helps!

[vikram_m]

Thanks for your reply @mayurr98 the same configuration is deployed which you have said above.

can you confirm if the sourcetype definition which we are doing is correct or do we need to have a bracket like <> when we want to define our sourcetype.

[aakwah]

Universal forwarder can not perform this parsing you have to use you need to put this in Heavy forwarder as mentioned in the above comment.