i have application logs which contain a message template in a json field (@mt) to convert other json fields into a human readable message. The content of that field will look like
{RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.
the text in the {} corresponds to the keys in key value pairs in the rest of the json
i made this eval
eval message="\"".replace(replace(spath(_raw, "@mt"),"{", "\"."),"}",".\"")."\""
this returns
message="".RequestProtocol." ".RequestMethod." ".RequestPath." responded ".StatusCode.""
it dispays the key names rather than the values
in that same search i have
eval test=" ".RequestProtocol." ".RequestMethod." ".RequestPath." responded ".StatusCode.""
that returns
test=HTTP/1.1 POST /path/to/thing responded 202
this returns the values.
how can i create an eval that forms the message from the template?
i would love to make this part of the sourcetype
This is a great question and a use case I'm surprised I haven't seen before!
| makeresults
| eval @mt="{RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.", RequestProtocol="HTTP/1.1", RequestMethod="POST", RequestPath="/path/to/thing", StatusCode="202"
| rex field=@mt max_match=0 "{(?<replacement_fields>[^}]+)"
| eval message='@mt'
| foreach * [ | eval message=coalesce(replace(message, "{".mvindex(replacement_fields, mvfind(replacement_fields, "<<FIELD>>"))."}", '<<FIELD>>'), message) ]
@mt | RequestMethod | RequestPath | RequestProtocol | StatusCode | _time | message | replacement_fields |
RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}. | POST | /path/to/thing | HTTP/1.1 | 202 | 2021-03-06 14:31:00 PM | HTTP/1.1 POST /path/to/thing responded 202. | RequestProtocol RequestMethod RequestPath StatusCode |
I've included the coalesce(..., message) function to prevent null values from clobbering the message field during the foreach loop.
If your format strings (which appear .NET-like) support escaping braces with double-braces, i.e. {{ and }}, you'll need to modify the rex command's regular expression slightly. One possibility is "(?<!{){(?!{)(?<replacement_fields>(?:{{|}}|[^}])+)(?<!})}(?!})" but we'd need to understand the rules of your format strings to validate.
Edit: If you have control over the output, I would suggest replacing format strings with an identifier, e.g. "event_id":"123", and creating a lookup table that contains message formats:
event_id | format_string |
123 | {RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}. |
Incidentally, this is how messages and localization strings (with the addition of a locale field) work in most systems. It allows you to decouple your data (the event fields) from your presentation (the message).
This Worked great using
This is a great question and a use case I'm surprised I haven't seen before!
| makeresults
| eval @mt="{RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.", RequestProtocol="HTTP/1.1", RequestMethod="POST", RequestPath="/path/to/thing", StatusCode="202"
| rex field=@mt max_match=0 "{(?<replacement_fields>[^}]+)"
| eval message='@mt'
| foreach * [ | eval message=coalesce(replace(message, "{".mvindex(replacement_fields, mvfind(replacement_fields, "<<FIELD>>"))."}", '<<FIELD>>'), message) ]
@mt | RequestMethod | RequestPath | RequestProtocol | StatusCode | _time | message | replacement_fields |
RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}. | POST | /path/to/thing | HTTP/1.1 | 202 | 2021-03-06 14:31:00 PM | HTTP/1.1 POST /path/to/thing responded 202. | RequestProtocol RequestMethod RequestPath StatusCode |
I've included the coalesce(..., message) function to prevent null values from clobbering the message field during the foreach loop.
If your format strings (which appear .NET-like) support escaping braces with double-braces, i.e. {{ and }}, you'll need to modify the rex command's regular expression slightly. One possibility is "(?<!{){(?!{)(?<replacement_fields>(?:{{|}}|[^}])+)(?<!})}(?!})" but we'd need to understand the rules of your format strings to validate.
Edit: If you have control over the output, I would suggest replacing format strings with an identifier, e.g. "event_id":"123", and creating a lookup table that contains message formats:
event_id | format_string |
123 | {RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}. |
Incidentally, this is how messages and localization strings (with the addition of a locale field) work in most systems. It allows you to decouple your data (the event fields) from your presentation (the message).