Getting a bit more complex but I think something like this will get you started.
| rex field=_raw "\]\s+(?<log_level>[A-Z]+)\s+(?<log_type>[^\s]+)\s+\-"
| rex field=_raw "\-\s+\([^\)]+\)\s+(?<message_detail>(?:.*(?:[\r\n]+)*)*)"
| rex field=_raw "user\s+\'(?<user>[^\']+)\'"
| rex field=_raw "client\s+(?<client>\d{1,3}(?:\.\d{1,3}){3})\:(?<port>\d+)"
| rex field=_raw "\[TID\:(?<tid>[^\]]+)\]"
| rex field=_raw "\]\s+[A-Z]+\s+[a-zA-Z0-9]+(?:\.[a-zA-Z0-9]+)*\s+\-\s+(?<message>.*)"
| rex field=_raw "(?i)cannot\s+open\s+(?<service>[^\s]+)\s+service\s+on\s+computer\s+\'(?<computer>[^\']+)\'"
``` log_type="Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner" ```
| rex field=_raw "(?i)\s+\-\s+(?<app>.*?)\s+unable\s+to\s+connect\s+to\s+(?<hostname>[^\s]+)\s+with\s+exception\s+(?<exception_type>[^\:]+)\:\s+(?<exception_message>.*)"
``` log_type="Thycotic.Discovery.Sources.Scanners.Dependency.ApplicationPoolScanner" error_message ```
| rex field=_raw "(?i)\s+\-\s+retrieving\s+the\s+(?<class>[^\s]+)\s+class\s+factory\s+for\s+remote\s+component\s+with\s+clsid\s+\{(?<clsid>[^\}]+)\}\s+from\s+machine\s+(?<hostname>[^\s]+)\s+failed\s+due\s+to\s+the\s+following\s+error\:\s+(?<error_code>[^\s]+)"
| rex field=_raw "(?i)(?<exception_type>\w+(\.\w+)*exception)\:\s+(?<exception_message>.*)"
Since the evals to standardize messages probably isn't all that scalable with these logs, it may make sense to set up eventtyping around them to assist with any sort of analysis. Once you have eventtypes set up that can assist with targeting specific type of events and finding patterns that you can develop regex for and pull out all the necessary fields.
screenshot below for reference of what the above regex should do.
Something like this maybe?
<base_search>
| rex field=_raw "\]\s+(?<log_level>[A-Z]+)\s+(?<log_type>[^\s]+)\s+\-\s+(?<message>.*?)\s+\-"
| rex field=_raw "\-\s+\([^\)]+\)\s+(?<message_detail>(?:.*(?:[\r\n]+)*)*)"
| rex field=_raw "user\s+\'(?<user>[^\']+)\'"
| rex field=_raw "client\s+(?<client>\d{1,3}(?:\.\d{1,3}){3})\:(?<port>\d+)"
| rex field=_raw "\[TID\:(?<tid>[^\]]+)\]"
Not familiar with the data but based on the 2 sample logs you provided was able to identify a few fields that I think would be useful. You can see from the screenshot below that these fields appear to be successfully extracted.
And to standardize the messages (at least for these examples) you can add an eval like this after the field extractions.
| eval
standardized_message=replace(replace(message, "client\s+(\d{1,3}(?:\.\d{1,3}){3})\:(\d+)", "client"), "user\s+\'([^\']+)\'", "user")You can see in the screenshot that the field 'standardized_message' removes any variable data so you can do some sort of aggregate analysis against the standardized message itself if needed.
