topic Re: Data masking using heavy forwarders in Getting Data In

Data masking using heavy forwarders

Venkat_16 — Tue, 29 Sep 2020 17:01:16 GMT

Been trying to mask data before indexing into indexer using heavy forwarders. below is the log sample and data am trying to mask

JSESSIONID=SD1SL10FF3ADFF3" to JSESSIONID=#######FF3ADFF3"

189.222.1.46 - - [24/Jul/2014:11:27:00] "GET /flower_store/product.screen?product_id=RP-SN-01 HTTP/1.1" 200 10897 "http://mystore.splunk.com/flower_store/category.screen?category_id=BALLOONS&JSESSIONID=SD1SL10FF3ADFF3" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 527 3006

10.2.91.38 - - [24/Jul/2014:11:28:00] "POST /flower_store/j_signon_check HTTP/1.1" 302 309
"http://mystore.splunk.com/flower_store/enter_order_information.screen&JSESSIONID=SD1SL10FF3ADFF3" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 3441 2576

192.0.1.38 - - [24/Jul/2014:11:28:15] "GET /flower_store/images/cat3.gif HTTP/1.1" 200 5024 "http://mystore.splunk.com/flower_store/item.screen?item_id=EST-21&JSESSIONID=SD1SL10FF3ADFF3" "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10" 4323 3071

below is my props.conf and transforms.conf:

[access_common]
TRANSFORMS-anonymize = session-anonymizer

[session-anonymizer]
REGEX = (?m)^(.)JSESSIONID=\w{2}\d\w{2}\d{2}(\w+.)
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw

Kindly advice....i do not see and changes in fields after applying this configuration

Re: Data masking using heavy forwarders

harsmarvania57 — Tue, 28 Nov 2017 08:28:50 GMT

Hi @Venkat_16,

Please use below configuration in transforms.conf on Heavy forwarder.

[session-anonymizer]
REGEX = (?m)^(.*)JSESSIONID=.*((?=\"\s\").*)$
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw

After changing above configuration please reload configuration using https://<HF FQDN>:8000/debug/refresh OR restart splunk on heavy forwarder.

I hope this helps.

Thanks,
Harshil

Re: Data masking using heavy forwarders

Venkat_16 — Tue, 28 Nov 2017 10:30:19 GMT

Thanks for your answer @harsmarvania57. But it didnt work 😞

[access_common]
TRANSFORMS-anonymize = session-anonymizer

[session-anonymizer]
REGEX = (?m)^(.)JSESSIONID=\w{2}\d\w{2}\d{2}(\w+.)
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw

i restarted heavy forwarder also....logs are getting indexed with out any masking

Re: Data masking using heavy forwarders

harsmarvania57 — Tue, 28 Nov 2017 10:32:23 GMT

Looks like you are still using your old configuration, please use configuration which I have provided.

Re: Data masking using heavy forwarders

Venkat_16 — Tue, 28 Nov 2017 10:35:39 GMT

sorry it was copy paste error:

[session-anonymizer]
REGEX = (?m)^(.)JSESSIONID=.((?=\"\s\").*)$
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw
is the one am using

Re: Data masking using heavy forwarders

harsmarvania57 — Tue, 28 Nov 2017 10:48:16 GMT

Still this configuration is wrong, you can see my regex has (?m)^(.*)
and configuration which you pasted starts with (?m)^(.)

Re: Data masking using heavy forwarders

Venkat_16 — Tue, 28 Nov 2017 12:03:22 GMT

Thanks alot for helping patiently but still it didnt work:

[access_common]
TRANSFORMS-anonymize = session-anonymizer

[session-anonymizer]
REGEX = (?m)^(.)JSESSIONID=.((?=\"\s\").*)$
FORMAT = $1JSESSIONID=#######$2
DEST_KEY = _raw

Re: Data masking using heavy forwarders

harsmarvania57 — Tue, 28 Nov 2017 12:10:47 GMT

Please refer this https://regex101.com/r/CftIqK/1, regex is working perfectly fine. Can you please provide your configuration in Code Sample format because your REGEX is still wrong, you can see 101010 button when you type comment/answer please use that and paste your transforms.conf configuration.

Re: Data masking using heavy forwarders

cpetterborg — Tue, 28 Nov 2017 17:08:05 GMT

All you need in this case is the following line in your props.conf file (no transforms.conf config needed) for the sourcetype:

SEDCMD-hidesessionid = s/JSESSIONID=\w{2}\d\w{2}\d{2}/JSESSIONID=#######/

Re: Data masking using heavy forwarders

woodcock — Wed, 29 Nov 2017 02:05:17 GMT

This works:

| makeresults 
| eval raw="189.222.1.46 - - [24/Jul/2014:11:27:00] \"GET /flower_store/product.screen?product_id=RP-SN-01 HTTP/1.1\" 200 10897 \"http://mystore.splunk.com/flower_store/category.screen?category_id=BALLOONS&JSESSIONID=SD1SL10FF3ADFF3\" \"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10\" 527 3006:::10.2.91.38 - - [24/Jul/2014:11:28:00] \"POST /flower_store/j_signon_check HTTP/1.1\" 302 309 
\"http://mystore.splunk.com/flower_store/enter_order_information.screen&JSESSIONID=SD1SL10FF3ADFF3\" \"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10\" 3441 2576:::192.0.1.38 - - [24/Jul/2014:11:28:15] \"GET /flower_store/images/cat3.gif HTTP/1.1\" 200 5024 \"http://mystore.splunk.com/flower_store/item.screen?item_id=EST-21&JSESSIONID=SD1SL10FF3ADFF3\" \"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.0.10) Gecko/20070223 CentOS/1.5.0.10-0.1.el4.centos Firefox/1.5.0.10\" 4323 3071"
| makemv delim=":::" raw
| mvexpand raw
| rename raw AS _raw
| rex mode=sed "s/JSESSIONID=\w{2}\d\w{2}\d{2}(\w+.)/JSESSIONID=#######\1/"

So you can use this:

 SEDCMD-session-anonymizer = s/JSESSIONID=\w{2}\d\w{2}\d{2}(\w+.)/JSESSIONID=#######\1/