requirements:
find and save sensitive data fields from logs
Save log snippet around sensitive data field
Remove duplicates for mule apps and sensitive data field
Create table showing mule app name, sensitive data, and log snippet
is there a way to improve the search query so I don't have to duplicate the rex commands every time I need to add a new sensitive data value?
(app_name is an existing custom field)
index="prod" |rex field=_raw (?i)(?<birthDate>(birthDate))|rex field=_raw (?i)(?<dob>(dob)) |rex field=_raw (?i)(?<birthday>(birthday)) |rex field=_raw (?i)(?<birthDateLog>(birthDate).*?\w\W) |rex field=_raw (?i)(?<dobLog>(dob).*?\w\W) |rex field=_raw (?i)(?<birthdayLog>(birthday).*?\w\W)|eval SENSITIVE_DATA= mvappend(birthDate,dob,birthday) |eval SENSITIVE_DATA_LOWER=lower(SENSITIVE_DATA) | dedup app_name SENSITIVE_DATA_LOWER |eval SENSITIVE_DATA_LOG=mvappend(birthDateLog,dobLog,birthdayLog) |stats list(SENSITIVE_DATA_LOG) as SENSITIVE_DATA_LOG list(SENSITIVE_DATA_LOWER) as SENSITIVE_DATA_LOWER by app_name | table app_name SENSITIVE_DATA_LOWER SENSITIVE_DATA_LOG
example output:
| app_name | SENSITIVE_DATA_LOWER | SENSITIVE_DATA_LOG |
| s-api | dob birthdate | dob: 01/01/2024 birthdate: 09-09-1999 |
| p-api | birthday | birthday: August 23, 2024 |
Well, you can try to make a compound regex containing some alternative branches.
Also you seem to have some XML-like structure there. If it's a valid XML, why not just parse the XML into fields and check for existence of specific fields?
I'm also not sure about the rest of the search but honestly speaking it's too late and I'm too tired at the moment to look into it.