How to improve readability with template formatted log messages in Splunk search?

[kinicky]

I have a dotnet application logging template formatted log messages with serilog library and since everything is in JSON format they are great to filter my results when I know the fields to use but I am having a hard time just to read logs when I dont know the fields available.

So for example, the application might log things like:
Log.Information("Just got a request {request} in endpoint {endpoint} with {httpMethod}", request,endpoint, httpMethod);

And in Splunk I will see something like:

{
    "msg": {
        "@mt": "Just got a request {request} in endpoint {endpoint} with {httpMethod}",
        "@sp": "11111",
        "request": "some_data",
        "endpoint": "some_url".
        "httpMethod": "POST"
    }
}

So this is awesome to create splunk queries using msg.request or msg.endpoint, but since the application logs pretty much everything using these message templates from serilog, when I am just doing investigations, I have a hard time in making readable results because everythig is hidden behind a placeholder.

I am trying to achieve something like in Splunk Search:

<some_guid> index=some_index | table _time msg.@mt 

and of course the msg.@mt will just give me the log line with the placeholders, but how can I just bring back the full log line in the table with the actual values?

[ITWhisperer]

Following on from @PickleRick suggestion, to avoid the @mt issue, you could do something like this

| spath msg.@mt output=mt
| rex field=mt max_match=0 "{(?<templates>[^}]+)}"
| foreach mode=multivalue templates 
     [ | eval mt=replace(mt,"{".<<ITEM>>."}", json_extract(_raw,"msg.".<<ITEM>>)) ]

[PickleRick]

Ugh. That is one ugly cookie.

On the one hand it's indeed very convenient to search if you know what to look for. It's the opposite of parsing the fields out of your logs - you already have them so you're saving yourself a lot of work and don't have to create extractions for various message formats.

But on the other hand - I don't recall any function which gives you this kind of formatting. You could try using foreach in multivalue mode (after extracting the placeholders from the template). Since your original data is json you can try using json_extract to find the value for replacing your placeholders.

In a general case you can't just get a value of a field by its name stored in another field. But since your data is a json, you might be able to pull off something like (just a general idea, might have typos and such

| rex field='@mt' max_match=0 "{(?<templates>[^{]+)}"
| foreach mode=multivalue templates 
     [ | eval '@mt'=replace('@mt',"{".<<ITEM>>."}", json_extract(_raw,"msg.".<<ITEM>>) ]

I'm not even sure you can have a field named @MT - you might need to rename it.