Splunk Search

dynamic eval statement from a template contained in the event data

lbruhns
Explorer

i have application logs which contain a message template in a json field (@mt) to convert other json fields into a human readable message. The content of that field will look like

{RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.

the text in the {} corresponds to the keys in key value pairs in the rest of the json

i made this eval

eval message="\"".replace(replace(spath(_raw, "@mt"),"{", "\"."),"}",".\"")."\""

this returns

message="".RequestProtocol." ".RequestMethod." ".RequestPath." responded ".StatusCode.""

it dispays the key names rather than the values

in that same search i have 

eval test=" ".RequestProtocol." ".RequestMethod." ".RequestPath." responded ".StatusCode.""

that returns

test=HTTP/1.1 POST /path/to/thing responded 202

this returns the values. 

how can i create an eval that forms the message from the template?

i would love to make this part of the sourcetype 

Labels (1)
0 Karma
1 Solution

tscroggins
Builder

@lbruhns 

This is a great question and a use case I'm surprised I haven't seen before!

 

 

 

| makeresults
| eval @mt="{RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.", RequestProtocol="HTTP/1.1", RequestMethod="POST", RequestPath="/path/to/thing", StatusCode="202"
| rex field=@mt max_match=0 "{(?<replacement_fields>[^}]+)"
| eval message='@mt'
| foreach * [ | eval message=coalesce(replace(message, "{".mvindex(replacement_fields, mvfind(replacement_fields, "<<FIELD>>"))."}", '<<FIELD>>'), message) ]

 

 

 

 

@mtRequestMethodRequestPathRequestProtocolStatusCode_timemessagereplacement_fields
RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.POST/path/to/thingHTTP/1.12022021-03-06 14:31:00 PMHTTP/1.1 POST /path/to/thing responded 202.RequestProtocol
RequestMethod
RequestPath
StatusCode

 

I've included the coalesce(..., message) function to prevent null values from clobbering the message field during the foreach loop.

If your format strings (which appear .NET-like) support escaping braces with double-braces, i.e. {{ and }}, you'll need to modify the rex command's regular expression slightly. One possibility is "(?<!{){(?!{)(?<replacement_fields>(?:{{|}}|[^}])+)(?<!})}(?!})" but we'd need to understand the rules of your format strings to validate.

Edit: If you have control over the output, I would suggest replacing format strings with an identifier, e.g. "event_id":"123", and creating a lookup table that contains message formats:

event_idformat_string
123{RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.

 

Incidentally, this is how messages and localization strings (with the addition of a locale field) work in most systems. It allows you to decouple your data (the event fields) from your presentation (the message).

View solution in original post

0 Karma

lbruhns
Explorer

This Worked great using 

rex field=@mt max_match=0 "{(?<replacement_fields>[^}]+)"
| eval message='@mt'
| foreach * [ | eval message=coalesce(replace(message, "{".mvindex(replacement_fields, mvfind(replacement_fields, "<<FIELD>>"))."}", '<<FIELD>>'), message) ]
 
(?<!\{){(?!\{)(?<replacement_fields>[^}]+) this regex managed the escaped brackets
 
now i'll try to somehow make this part of the sourcetype, more likely i'll make a macro. the lookup table solution is not desireable, with the template in the log it gives the developers complete control to change the message to meet their needs without having to have a process for adding or appending templates
0 Karma

tscroggins
Builder

@lbruhns 

This is a great question and a use case I'm surprised I haven't seen before!

 

 

 

| makeresults
| eval @mt="{RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.", RequestProtocol="HTTP/1.1", RequestMethod="POST", RequestPath="/path/to/thing", StatusCode="202"
| rex field=@mt max_match=0 "{(?<replacement_fields>[^}]+)"
| eval message='@mt'
| foreach * [ | eval message=coalesce(replace(message, "{".mvindex(replacement_fields, mvfind(replacement_fields, "<<FIELD>>"))."}", '<<FIELD>>'), message) ]

 

 

 

 

@mtRequestMethodRequestPathRequestProtocolStatusCode_timemessagereplacement_fields
RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.POST/path/to/thingHTTP/1.12022021-03-06 14:31:00 PMHTTP/1.1 POST /path/to/thing responded 202.RequestProtocol
RequestMethod
RequestPath
StatusCode

 

I've included the coalesce(..., message) function to prevent null values from clobbering the message field during the foreach loop.

If your format strings (which appear .NET-like) support escaping braces with double-braces, i.e. {{ and }}, you'll need to modify the rex command's regular expression slightly. One possibility is "(?<!{){(?!{)(?<replacement_fields>(?:{{|}}|[^}])+)(?<!})}(?!})" but we'd need to understand the rules of your format strings to validate.

Edit: If you have control over the output, I would suggest replacing format strings with an identifier, e.g. "event_id":"123", and creating a lookup table that contains message formats:

event_idformat_string
123{RequestProtocol} {RequestMethod} {RequestPath} responded {StatusCode}.

 

Incidentally, this is how messages and localization strings (with the addition of a locale field) work in most systems. It allows you to decouple your data (the event fields) from your presentation (the message).

View solution in original post

0 Karma