How to mask a password in a log coming from an HTTP Event Collector

[dnavia29]

I am trying to mask a password that is inside a log coming from HTTP Event Collector.

I configure my props.conf with the following

[api-core]
TRANSFORMS-anonymize = password-anonymizer

and my transforms.conf like this
[password-anonymizer]
REGEX =
FORMAT = xxxxx
DEST_KEY = _raw

I want to mask the password that is inside this log but I can't find the way to make the regular expression for this.


{"api_id":"5e4d6034e4b0258f388e1dfe","app_type":"PRODUCTION","bytes_received":57,"response_body":"","client_id":"4b7eff29-39ca-4728-ba28-b8889308600d","billing":{"amount":0,"provider":"none","currency":"USD","model":"free","trial_period_days":0},"datetime":"2020-02-19T16:29:23.535Z","time_to_serve_request":23,"uri_path":"/public/human-resource/v1.0/users/password-reset","log_policy":"payload","endpoint_url":"N/A","product_id":"__INTERNAL_QS__","host":"127.0.0.1","client_ip":"10.181.37.19","app_id":"__INTERNAL_QS__:1.0.0:default","client_geoip":{},"request_protocol":"https","developer_org_id":"5ddfc086e4b0740304d6c3e0","transaction_id":"66306","immediate_client_ip":"10.181.37.19","product_name":"__INTERNAL_QS__","plan_name":"default","product_title":"","tags":["_geoip_lookup_failure"],"catalog_id":"5ddfc67ce4b0740304d6c427","space_name":[""],"api_name":"Authentication","org_id":"5ddfc086e4b0740304d6c3e0","plan_version":"1.0.0","status_code":"400 Bad Request","request_method":"PUT","developer_org_name":"public","http_user_agent":"Dalvik/2.1.0 (Linux; U; Android 7.0; Moto G (4) Build/NPJS25.93-14-8.1-9)","resource_path":"put","@version":"1","response_http_headers":[{"Server":"Microsoft-IIS/10.0"},{"transaction_id":"36370bf9-9239-4e8b-bc41-aae8ffc431c5"},{"timestamp":"2020-02-19T16:29:23Z"},{"channel-id":""},{"application":""},{"Itau-Client-Secret":""},{"Itau-Client-Id":""},{"X-Powered-By":"ASP.NET"},{"Date":"Wed, 19 Feb 2020 16:29:23 GMT"},{"X-Global-Transaction-ID":"ce91764b5e4d626300010302"},{"Access-Control-Expose-Headers":"APIm-Debug-Trans-Id, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-Global-Transaction-ID"},{"Access-Control-Allow-Origin":"*"},{"Access-Control-Allow-Methods":"PUT"}],"org_name":"public","latency_info":[{"task":"Start","started":0},{"task":"security-appID","started":7},{"task":"invoke","started":9}],"headers":{"http__ws_haprt_wlmversion":"-1","http_via":"1.1 AwAAAKsfL+8-","http_version":"HTTP/1.1","http_connection":"Keep-Alive","request_method":"POST","http_host":"localhost:9700","request_uri":"/_bulk","http_x_forwarded_server":"apimngdes.itauchile.cl","content_type":"text/plain","http_x_global_transaction_id":"ce91764b5e56ca470154b0f1","http_x_forwarded_host":"10.181.168.56:9443","http_x_forwarded_for":"10.181.168.63","request_path":"/_bulk","http_organization":"admin","http_x_client_ip":"127.0.0.1","content_length":"211346"},"catalog_name":"human-resource","product_version":"1.0.0","debug":[],"rateLimit":{"rate-limit":{"limit":"-1","count":"-1"},"rate-limit-1":{"limit":"-1","count":"-1"},"rate-limit-2":{"limit":"-1","count":"-1"},"per-minute":{"limit":"-1","count":"-1"}},"api_version":"v1","bytes_sent":0,"app_name":"__INTERNAL_QS__","gateway_geoip":{},"@timestamp":"2020-02-26T19:43:03.957Z","request_body":"{ \"password_new\":\"qwe123\", \"password_new_confirm\":\"qwe123\" }","request_http_headers":[{"Content-Type":"application/json"},{"Accept":"application/json"},{"charset":"utf-8"},{"authorization":"********sanitized********"},{"Itau-Client-Secret":"kO1yD5bJ2bX8dS8eR3pQ7mQ6cM0uO0aV6mX7dG5oP6xD4kD5uD"},{"Itau-Client-Id":"4b7eff29-39ca-4728-ba28-b8889308600d"},{"User-Agent":"Dalvik/2.1.0 (Linux; U; Android 7.0; Moto G (4) Build/NPJS25.93-14-8.1-9)"},{"Host":"clstgappd01v5.itauchile.cl"},{"Accept-Encoding":"gzip"},{"Content-Length":"57"},{"Via":"1.1 AQAAAKCPm9Q-"},{"X-Client-IP":"10.181.37.19"},{"X-Global-Transaction-ID":"ce91764b5e4d626300010302"}],"resource_id":"Authentication:v1:put:/v1.0/users/password-reset","gateway_ip":"10.181.168.63","space_id":[""],"plan_id":"__INTERNAL_QS__:1.0.0:default","developer_org_title":"undefined","query_string":[]}
Show syntax highlighted
host = 10.181.167.158:8088 host = 127.0.0.1request_body = { "password_new":"qwe123", "password_new_confirm":"qwe123" }source = http:api_connect_tokensourcetype = api-coreuri_path = /public/human-resource/v1.0/users/password-reset

I want to mask the password_new":"qwe123 to be password: xxxxxx`

Please your help with this

[starcher]

If you are posting to the JSON endpoint not the raw, there is no processing on the data for transforms to work. Best solution is don't send the password to begin with. Edit the sending source's send code to exclude it.

[dshpritz]

According to some of the HEC folks, even /event endpoint still engages the regexreplacement, just not the line breaking and timestamping processors. But this reaaaally needs to be documented.

[pruthvikrishnap]

in your case may be something like this
props.conf
[api-core]
TRANSFORMS-anonymize = password-anonymizer

transforms.conf like this
[password-anonymizer]
REGEX = s/(password_new(?:_confirm)?\?":\?")([^\"]+)/\1xxxxxx/g
FORMAT = $1Password:########,$2
DEST_KEY = _raw

[dnavia29]

thank you it really helped

[dshpritz · Accepted solution]

You would be better off using SEDCMD for this. Something like:

props.conf:

[api-core]
SEDCMD-password-anonymizer = s/(password_new(?:_confirm)?\\?":\s*\\?")([^\\"]+)/\1xxxxxx/g

[dnavia29]

Hello, thanks for the input.

I tried to test this directly in the command line creating a test.txt file with the log in it and running this command to prove the function but it is not working and masking the password.

sed -i '.bak' 's/\(password_new(?:_confirm\)?\\?":\\?")([^\\"]+)/\xxxxxx/g' test.txt

I am using '.bak' cause I have a Mac and it was returning an error regarding this, I tried to test it here first to see if it works before put in into splunk

[dshpritz]

Command line sed is not the same as Splunk's SEDCMD command. The regex would need to be modified. On a mac that would look something like like:

sed -E "s/(password_new(_confirm)?\\\\?\":\\\\?\\\\?\s*\")([^\\\"]+)/\1xxxxxx/g" < test.txt

[dnavia29]

It worked perfectly thank you very much