- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I am trying to mask a password that is inside a log coming from HTTP Event Collector.
I configure my props.conf with the following
[api-core]
TRANSFORMS-anonymize = password-anonymizer
and my transforms.conf like this
[password-anonymizer]
REGEX =
FORMAT = xxxxx
DEST_KEY = _raw
I want to mask the password that is inside this log but I can't find the way to make the regular expression for this.
{"api_id":"5e4d6034e4b0258f388e1dfe","app_type":"PRODUCTION","bytes_received":57,"response_body":"","client_id":"4b7eff29-39ca-4728-ba28-b8889308600d","billing":{"amount":0,"provider":"none","currency":"USD","model":"free","trial_period_days":0},"datetime":"2020-02-19T16:29:23.535Z","time_to_serve_request":23,"uri_path":"/public/human-resource/v1.0/users/password-reset","log_policy":"payload","endpoint_url":"N/A","product_id":"__INTERNAL_QS__","host":"127.0.0.1","client_ip":"10.181.37.19","app_id":"__INTERNAL_QS__:1.0.0:default","client_geoip":{},"request_protocol":"https","developer_org_id":"5ddfc086e4b0740304d6c3e0","transaction_id":"66306","immediate_client_ip":"10.181.37.19","product_name":"__INTERNAL_QS__","plan_name":"default","product_title":"","tags":["_geoip_lookup_failure"],"catalog_id":"5ddfc67ce4b0740304d6c427","space_name":[""],"api_name":"Authentication","org_id":"5ddfc086e4b0740304d6c3e0","plan_version":"1.0.0","status_code":"400 Bad Request","request_method":"PUT","developer_org_name":"public","http_user_agent":"Dalvik/2.1.0 (Linux; U; Android 7.0; Moto G (4) Build/NPJS25.93-14-8.1-9)","resource_path":"put","@version":"1","response_http_headers":[{"Server":"Microsoft-IIS/10.0"},{"transaction_id":"36370bf9-9239-4e8b-bc41-aae8ffc431c5"},{"timestamp":"2020-02-19T16:29:23Z"},{"channel-id":""},{"application":""},{"Itau-Client-Secret":""},{"Itau-Client-Id":""},{"X-Powered-By":"ASP.NET"},{"Date":"Wed, 19 Feb 2020 16:29:23 GMT"},{"X-Global-Transaction-ID":"ce91764b5e4d626300010302"},{"Access-Control-Expose-Headers":"APIm-Debug-Trans-Id, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-Global-Transaction-ID"},{"Access-Control-Allow-Origin":"*"},{"Access-Control-Allow-Methods":"PUT"}],"org_name":"public","latency_info":[{"task":"Start","started":0},{"task":"security-appID","started":7},{"task":"invoke","started":9}],"headers":{"http__ws_haprt_wlmversion":"-1","http_via":"1.1 AwAAAKsfL+8-","http_version":"HTTP/1.1","http_connection":"Keep-Alive","request_method":"POST","http_host":"localhost:9700","request_uri":"/_bulk","http_x_forwarded_server":"apimngdes.itauchile.cl","content_type":"text/plain","http_x_global_transaction_id":"ce91764b5e56ca470154b0f1","http_x_forwarded_host":"10.181.168.56:9443","http_x_forwarded_for":"10.181.168.63","request_path":"/_bulk","http_organization":"admin","http_x_client_ip":"127.0.0.1","content_length":"211346"},"catalog_name":"human-resource","product_version":"1.0.0","debug":[],"rateLimit":{"rate-limit":{"limit":"-1","count":"-1"},"rate-limit-1":{"limit":"-1","count":"-1"},"rate-limit-2":{"limit":"-1","count":"-1"},"per-minute":{"limit":"-1","count":"-1"}},"api_version":"v1","bytes_sent":0,"app_name":"__INTERNAL_QS__","gateway_geoip":{},"@timestamp":"2020-02-26T19:43:03.957Z","request_body":"{ \"password_new\":\"qwe123\", \"password_new_confirm\":\"qwe123\" }","request_http_headers":[{"Content-Type":"application/json"},{"Accept":"application/json"},{"charset":"utf-8"},{"authorization":"********sanitized********"},{"Itau-Client-Secret":"kO1yD5bJ2bX8dS8eR3pQ7mQ6cM0uO0aV6mX7dG5oP6xD4kD5uD"},{"Itau-Client-Id":"4b7eff29-39ca-4728-ba28-b8889308600d"},{"User-Agent":"Dalvik/2.1.0 (Linux; U; Android 7.0; Moto G (4) Build/NPJS25.93-14-8.1-9)"},{"Host":"clstgappd01v5.itauchile.cl"},{"Accept-Encoding":"gzip"},{"Content-Length":"57"},{"Via":"1.1 AQAAAKCPm9Q-"},{"X-Client-IP":"10.181.37.19"},{"X-Global-Transaction-ID":"ce91764b5e4d626300010302"}],"resource_id":"Authentication:v1:put:/v1.0/users/password-reset","gateway_ip":"10.181.168.63","space_id":[""],"plan_id":"__INTERNAL_QS__:1.0.0:default","developer_org_title":"undefined","query_string":[]}
Show syntax highlighted
host = 10.181.167.158:8088 host = 127.0.0.1request_body = { "password_new":"qwe123", "password_new_confirm":"qwe123" }source = http:api_connect_tokensourcetype = api-coreuri_path = /public/human-resource/v1.0/users/password-reset
I want to mask the password_new":"qwe123 to be password: xxxxxx
`
Please your help with this
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You would be better off using SEDCMD for this. Something like:
props.conf:
[api-core]
SEDCMD-password-anonymizer = s/(password_new(?:_confirm)?\\?":\s*\\?")([^\\"]+)/\1xxxxxx/g
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
If you are posting to the JSON endpoint not the raw, there is no processing on the data for transforms to work. Best solution is don't send the password to begin with. Edit the sending source's send code to exclude it.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
According to some of the HEC folks, even /event endpoint still engages the regexreplacement, just not the line breaking and timestamping processors. But this reaaaally needs to be documented.
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
in your case may be something like this
props.conf
[api-core]
TRANSFORMS-anonymize = password-anonymizer
transforms.conf like this
[password-anonymizer]
REGEX = s/(password_new(?:_confirm)?\?":\?")([^\"]+)/\1xxxxxx/g
FORMAT = $1Password:########,$2
DEST_KEY = _raw
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
thank you it really helped
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
You would be better off using SEDCMD for this. Something like:
props.conf:
[api-core]
SEDCMD-password-anonymizer = s/(password_new(?:_confirm)?\\?":\s*\\?")([^\\"]+)/\1xxxxxx/g
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hello, thanks for the input.
I tried to test this directly in the command line creating a test.txt file with the log in it and running this command to prove the function but it is not working and masking the password.
sed -i '.bak' 's/\(password_new(?:_confirm\)?\\?":\\?")([^\\"]+)/\xxxxxx/g' test.txt
I am using '.bak' cause I have a Mac and it was returning an error regarding this, I tried to test it here first to see if it works before put in into splunk
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Command line sed is not the same as Splunk's SEDCMD command. The regex would need to be modified. On a mac that would look something like like:
sed -E "s/(password_new(_confirm)?\\\\?\":\\\\?\\\\?\s*\")([^\\\"]+)/\1xxxxxx/g" < test.txt
- Mark as New
- Bookmark Message
- Subscribe to Message
- Mute Message
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
It worked perfectly thank you very much