Splunk Search

How to mask a password in a log coming from an HTTP Event Collector

dnavia29
New Member

I am trying to mask a password that is inside a log coming from HTTP Event Collector.

I configure my props.conf with the following

[api-core]
TRANSFORMS-anonymize = password-anonymizer

and my transforms.conf like this
[password-anonymizer]
REGEX =
FORMAT = xxxxx
DEST_KEY = _raw

I want to mask the password that is inside this log but I can't find the way to make the regular expression for this.


{"api_id":"5e4d6034e4b0258f388e1dfe","app_type":"PRODUCTION","bytes_received":57,"response_body":"","client_id":"4b7eff29-39ca-4728-ba28-b8889308600d","billing":{"amount":0,"provider":"none","currency":"USD","model":"free","trial_period_days":0},"datetime":"2020-02-19T16:29:23.535Z","time_to_serve_request":23,"uri_path":"/public/human-resource/v1.0/users/password-reset","log_policy":"payload","endpoint_url":"N/A","product_id":"__INTERNAL_QS__","host":"127.0.0.1","client_ip":"10.181.37.19","app_id":"__INTERNAL_QS__:1.0.0:default","client_geoip":{},"request_protocol":"https","developer_org_id":"5ddfc086e4b0740304d6c3e0","transaction_id":"66306","immediate_client_ip":"10.181.37.19","product_name":"__INTERNAL_QS__","plan_name":"default","product_title":"","tags":["_geoip_lookup_failure"],"catalog_id":"5ddfc67ce4b0740304d6c427","space_name":[""],"api_name":"Authentication","org_id":"5ddfc086e4b0740304d6c3e0","plan_version":"1.0.0","status_code":"400 Bad Request","request_method":"PUT","developer_org_name":"public","http_user_agent":"Dalvik/2.1.0 (Linux; U; Android 7.0; Moto G (4) Build/NPJS25.93-14-8.1-9)","resource_path":"put","@version":"1","response_http_headers":[{"Server":"Microsoft-IIS/10.0"},{"transaction_id":"36370bf9-9239-4e8b-bc41-aae8ffc431c5"},{"timestamp":"2020-02-19T16:29:23Z"},{"channel-id":""},{"application":""},{"Itau-Client-Secret":""},{"Itau-Client-Id":""},{"X-Powered-By":"ASP.NET"},{"Date":"Wed, 19 Feb 2020 16:29:23 GMT"},{"X-Global-Transaction-ID":"ce91764b5e4d626300010302"},{"Access-Control-Expose-Headers":"APIm-Debug-Trans-Id, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-Global-Transaction-ID"},{"Access-Control-Allow-Origin":"*"},{"Access-Control-Allow-Methods":"PUT"}],"org_name":"public","latency_info":[{"task":"Start","started":0},{"task":"security-appID","started":7},{"task":"invoke","started":9}],"headers":{"http__ws_haprt_wlmversion":"-1","http_via":"1.1 AwAAAKsfL+8-","http_version":"HTTP/1.1","http_connection":"Keep-Alive","request_method":"POST","http_host":"localhost:9700","request_uri":"/_bulk","http_x_forwarded_server":"apimngdes.itauchile.cl","content_type":"text/plain","http_x_global_transaction_id":"ce91764b5e56ca470154b0f1","http_x_forwarded_host":"10.181.168.56:9443","http_x_forwarded_for":"10.181.168.63","request_path":"/_bulk","http_organization":"admin","http_x_client_ip":"127.0.0.1","content_length":"211346"},"catalog_name":"human-resource","product_version":"1.0.0","debug":[],"rateLimit":{"rate-limit":{"limit":"-1","count":"-1"},"rate-limit-1":{"limit":"-1","count":"-1"},"rate-limit-2":{"limit":"-1","count":"-1"},"per-minute":{"limit":"-1","count":"-1"}},"api_version":"v1","bytes_sent":0,"app_name":"__INTERNAL_QS__","gateway_geoip":{},"@timestamp":"2020-02-26T19:43:03.957Z","request_body":"{ \"password_new\":\"qwe123\", \"password_new_confirm\":\"qwe123\" }","request_http_headers":[{"Content-Type":"application/json"},{"Accept":"application/json"},{"charset":"utf-8"},{"authorization":"********sanitized********"},{"Itau-Client-Secret":"kO1yD5bJ2bX8dS8eR3pQ7mQ6cM0uO0aV6mX7dG5oP6xD4kD5uD"},{"Itau-Client-Id":"4b7eff29-39ca-4728-ba28-b8889308600d"},{"User-Agent":"Dalvik/2.1.0 (Linux; U; Android 7.0; Moto G (4) Build/NPJS25.93-14-8.1-9)"},{"Host":"clstgappd01v5.itauchile.cl"},{"Accept-Encoding":"gzip"},{"Content-Length":"57"},{"Via":"1.1 AQAAAKCPm9Q-"},{"X-Client-IP":"10.181.37.19"},{"X-Global-Transaction-ID":"ce91764b5e4d626300010302"}],"resource_id":"Authentication:v1:put:/v1.0/users/password-reset","gateway_ip":"10.181.168.63","space_id":[""],"plan_id":"__INTERNAL_QS__:1.0.0:default","developer_org_title":"undefined","query_string":[]}
Show syntax highlighted
host = 10.181.167.158:8088 host = 127.0.0.1request_body = { "password_new":"qwe123", "password_new_confirm":"qwe123" }source = http:api_connect_tokensourcetype = api-coreuri_path = /public/human-resource/v1.0/users/password-reset

I want to mask the password_new":"qwe123 to be password: xxxxxx`

Please your help with this

Tags (1)
0 Karma
1 Solution

dshpritz
SplunkTrust
SplunkTrust

You would be better off using SEDCMD for this. Something like:

props.conf:

[api-core]
SEDCMD-password-anonymizer = s/(password_new(?:_confirm)?\\?":\s*\\?")([^\\"]+)/\1xxxxxx/g

View solution in original post

0 Karma

starcher
SplunkTrust
SplunkTrust

If you are posting to the JSON endpoint not the raw, there is no processing on the data for transforms to work. Best solution is don't send the password to begin with. Edit the sending source's send code to exclude it.

0 Karma

dshpritz
SplunkTrust
SplunkTrust

According to some of the HEC folks, even /event endpoint still engages the regexreplacement, just not the line breaking and timestamping processors. But this reaaaally needs to be documented.

0 Karma

pruthvikrishnap
Contributor

in your case may be something like this
props.conf
[api-core]
TRANSFORMS-anonymize = password-anonymizer

transforms.conf like this
[password-anonymizer]
REGEX = s/(password_new(?:_confirm)?\?":\?")([^\"]+)/\1xxxxxx/g
FORMAT = $1Password:########,$2
DEST_KEY = _raw

0 Karma

dnavia29
New Member

thank you it really helped

0 Karma

dshpritz
SplunkTrust
SplunkTrust

You would be better off using SEDCMD for this. Something like:

props.conf:

[api-core]
SEDCMD-password-anonymizer = s/(password_new(?:_confirm)?\\?":\s*\\?")([^\\"]+)/\1xxxxxx/g
0 Karma

dnavia29
New Member

Hello, thanks for the input.

I tried to test this directly in the command line creating a test.txt file with the log in it and running this command to prove the function but it is not working and masking the password.

sed -i '.bak' 's/\(password_new(?:_confirm\)?\\?":\\?")([^\\"]+)/\xxxxxx/g' test.txt

I am using '.bak' cause I have a Mac and it was returning an error regarding this, I tried to test it here first to see if it works before put in into splunk

0 Karma

dshpritz
SplunkTrust
SplunkTrust

Command line sed is not the same as Splunk's SEDCMD command. The regex would need to be modified. On a mac that would look something like like:

sed -E "s/(password_new(_confirm)?\\\\?\":\\\\?\\\\?\s*\")([^\\\"]+)/\1xxxxxx/g" < test.txt
0 Karma

dnavia29
New Member

It worked perfectly thank you very much

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...