topic Re: How to mask a password in a log coming from an HTTP Event Collector in Splunk Search

How to mask a password in a log coming from an HTTP Event Collector

dnavia29 — Wed, 26 Feb 2020 20:10:38 GMT

I am trying to mask a password that is inside a log coming from HTTP Event Collector.

I configure my props.conf with the following

[api-core]
TRANSFORMS-anonymize = password-anonymizer

and my transforms.conf like this
[password-anonymizer]
REGEX =
FORMAT = xxxxx
DEST_KEY = _raw

I want to mask the password that is inside this log but I can't find the way to make the regular expression for this.

{"api_id":"5e4d6034e4b0258f388e1dfe","app_type":"PRODUCTION","bytes_received":57,"response_body":"","client_id":"4b7eff29-39ca-4728-ba28-b8889308600d","billing":{"amount":0,"provider":"none","currency":"USD","model":"free","trial_period_days":0},"datetime":"2020-02-19T16:29:23.535Z","time_to_serve_request":23,"uri_path":"/public/human-resource/v1.0/users/password-reset","log_policy":"payload","endpoint_url":"N/A","product_id":"__INTERNAL_QS__","host":"127.0.0.1","client_ip":"10.181.37.19","app_id":"__INTERNAL_QS__:1.0.0:default","client_geoip":{},"request_protocol":"https","developer_org_id":"5ddfc086e4b0740304d6c3e0","transaction_id":"66306","immediate_client_ip":"10.181.37.19","product_name":"__INTERNAL_QS__","plan_name":"default","product_title":"","tags":["_geoip_lookup_failure"],"catalog_id":"5ddfc67ce4b0740304d6c427","space_name":[""],"api_name":"Authentication","org_id":"5ddfc086e4b0740304d6c3e0","plan_version":"1.0.0","status_code":"400 Bad Request","request_method":"PUT","developer_org_name":"public","http_user_agent":"Dalvik/2.1.0 (Linux; U; Android 7.0; Moto G (4) Build/NPJS25.93-14-8.1-9)","resource_path":"put","@version":"1","response_http_headers":[{"Server":"Microsoft-IIS/10.0"},{"transaction_id":"36370bf9-9239-4e8b-bc41-aae8ffc431c5"},{"timestamp":"2020-02-19T16:29:23Z"},{"channel-id":""},{"application":""},{"Itau-Client-Secret":""},{"Itau-Client-Id":""},{"X-Powered-By":"ASP.NET"},{"Date":"Wed, 19 Feb 2020 16:29:23 GMT"},{"X-Global-Transaction-ID":"ce91764b5e4d626300010302"},{"Access-Control-Expose-Headers":"APIm-Debug-Trans-Id, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-Global-Transaction-ID"},{"Access-Control-Allow-Origin":"*"},{"Access-Control-Allow-Methods":"PUT"}],"org_name":"public","latency_info":[{"task":"Start","started":0},{"task":"security-appID","started":7},{"task":"invoke","started":9}],"headers":{"http__ws_haprt_wlmversion":"-1","http_via":"1.1 AwAAAKsfL+8-","http_version":"HTTP/1.1","http_connection":"Keep-Alive","request_method":"POST","http_host":"localhost:9700","request_uri":"/_bulk","http_x_forwarded_server":"apimngdes.itauchile.cl","content_type":"text/plain","http_x_global_transaction_id":"ce91764b5e56ca470154b0f1","http_x_forwarded_host":"10.181.168.56:9443","http_x_forwarded_for":"10.181.168.63","request_path":"/_bulk","http_organization":"admin","http_x_client_ip":"127.0.0.1","content_length":"211346"},"catalog_name":"human-resource","product_version":"1.0.0","debug":[],"rateLimit":{"rate-limit":{"limit":"-1","count":"-1"},"rate-limit-1":{"limit":"-1","count":"-1"},"rate-limit-2":{"limit":"-1","count":"-1"},"per-minute":{"limit":"-1","count":"-1"}},"api_version":"v1","bytes_sent":0,"app_name":"__INTERNAL_QS__","gateway_geoip":{},"@timestamp":"2020-02-26T19:43:03.957Z","request_body":"{ \"password_new\":\"qwe123\", \"password_new_confirm\":\"qwe123\" }","request_http_headers":[{"Content-Type":"application/json"},{"Accept":"application/json"},{"charset":"utf-8"},{"authorization":"********sanitized********"},{"Itau-Client-Secret":"kO1yD5bJ2bX8dS8eR3pQ7mQ6cM0uO0aV6mX7dG5oP6xD4kD5uD"},{"Itau-Client-Id":"4b7eff29-39ca-4728-ba28-b8889308600d"},{"User-Agent":"Dalvik/2.1.0 (Linux; U; Android 7.0; Moto G (4) Build/NPJS25.93-14-8.1-9)"},{"Host":"clstgappd01v5.itauchile.cl"},{"Accept-Encoding":"gzip"},{"Content-Length":"57"},{"Via":"1.1 AQAAAKCPm9Q-"},{"X-Client-IP":"10.181.37.19"},{"X-Global-Transaction-ID":"ce91764b5e4d626300010302"}],"resource_id":"Authentication:v1:put:/v1.0/users/password-reset","gateway_ip":"10.181.168.63","space_id":[""],"plan_id":"__INTERNAL_QS__:1.0.0:default","developer_org_title":"undefined","query_string":[]} Show syntax highlighted host = 10.181.167.158:8088 host = 127.0.0.1request_body = { "password_new":"qwe123", "password_new_confirm":"qwe123" }source = http:api_connect_tokensourcetype = api-coreuri_path = /public/human-resource/v1.0/users/password-reset

I want to mask the password_new":"qwe123 to be password: xxxxxx`

Please your help with this

Re: How to mask a password in a log coming from an HTTP Event Collector

dshpritz — Thu, 27 Feb 2020 00:43:25 GMT

You would be better off using SEDCMD for this. Something like:

props.conf:

[api-core]
SEDCMD-password-anonymizer = s/(password_new(?:_confirm)?\\?":\s*\\?")([^\\"]+)/\1xxxxxx/g

Re: How to mask a password in a log coming from an HTTP Event Collector

pruthvikrishnap — Wed, 30 Sep 2020 04:25:05 GMT

in your case may be something like this
props.conf
[api-core]
TRANSFORMS-anonymize = password-anonymizer

transforms.conf like this
[password-anonymizer]
REGEX = s/(password_new(?:_confirm)?\?":\?")([^\"]+)/\1xxxxxx/g
FORMAT = $1Password:########,$2
DEST_KEY = _raw

Re: How to mask a password in a log coming from an HTTP Event Collector

starcher — Thu, 27 Feb 2020 13:38:14 GMT

If you are posting to the JSON endpoint not the raw, there is no processing on the data for transforms to work. Best solution is don't send the password to begin with. Edit the sending source's send code to exclude it.

Re: How to mask a password in a log coming from an HTTP Event Collector

dnavia29 — Thu, 27 Feb 2020 15:05:01 GMT

Hello, thanks for the input.

I tried to test this directly in the command line creating a test.txt file with the log in it and running this command to prove the function but it is not working and masking the password.

sed -i '.bak' 's/$password_new(?:_confirm$?\\?":\\?")([^\\"]+)/\xxxxxx/g' test.txt

I am using '.bak' cause I have a Mac and it was returning an error regarding this, I tried to test it here first to see if it works before put in into splunk

Re: How to mask a password in a log coming from an HTTP Event Collector

dshpritz — Thu, 27 Feb 2020 15:37:57 GMT

According to some of the HEC folks, even /event endpoint still engages the regexreplacement, just not the line breaking and timestamping processors. But this reaaaally needs to be documented.

Re: How to mask a password in a log coming from an HTTP Event Collector

dshpritz — Thu, 27 Feb 2020 16:28:57 GMT

Command line sed is not the same as Splunk's SEDCMD command. The regex would need to be modified. On a mac that would look something like like:

sed -E "s/(password_new(_confirm)?\\\\?\":\\\\?\\\\?\s*\")([^\\\"]+)/\1xxxxxx/g" < test.txt

Re: How to mask a password in a log coming from an HTTP Event Collector

dnavia29 — Thu, 27 Feb 2020 19:06:23 GMT

It worked perfectly thank you very much

Re: How to mask a password in a log coming from an HTTP Event Collector

dnavia29 — Thu, 27 Feb 2020 19:06:53 GMT

thank you it really helped