Filter a log using regex

[jose_sepulveda]

I need to filter a part of a log using regex, I have the following log

log: {dx.trace_id=xxxxx, dx.span_id=yyyyy, dx.trace_sampled=true}{"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

I need to remove this fragment from the answer

{dx.trace_id=xxxxx, dx.span_id=yyyyy, dx.trace_sampled=true}

so that the visible log is the following

log: {"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

There are also outputs where what I need to filter is presented with fewer fields or without fields, leaving it this way

log: {dx.trace_sampled=true}{"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

log: {}{"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

In these last two examples I still need to filter the following respectively

{dx.trace_sampled=true}

{}

So that the output is finally clean and leaves only what I need

log: {"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

I hope you can help me please

[gcusello]

Hi @jose_sepulveda,

at first: do you want to remove a part of your logs before indexing or at search time?

if at index time remember that in this way you change the format of your logs, so the add-ons could not work correctly!

anyway, why do you want to remove a part of your logs?

your request doesn't seem to be an obfuscation.

anyway, you can do this using the command SEDCMD in the props.con using a substitution regex like the following:

SEDCMD = s/([^\}]+\})(.*)/$2/g

Ciao.

Giuseppe

[jose_sepulveda]

Hi @gcusello 

I have a service developed in JAVA that is dockerized, a shared tomcat image is used that is adding these fragments to the service's output logs, which are the ones I really need to view in splunk. For this reason the response is no longer a valid Json and the visualization is presented as a string and I need to resolve that situation

[gcusello]

Hi @jose_sepulveda ,

ok, please, check my solution or the one from @ITWhisperer that's similar.

Ciao.

Giuseppe

[jose_sepulveda]

Hi @gcusello , @ITWhisperer 

When adding the regular expression indicated by @ITWhisperer  to the search, it still appears in the result, which I need to remove

[ITWhisperer]

This looks like a corrupted / non-standard version of JSON (It would be helpful for you to share the unformatted version of the log since that is what the rex will be working with!). Try something like this

| rex mode=sed "s/\"log\": {[^}]*}/\"log\": {}/g"

[ITWhisperer]

| rex mode=sed "s/log: {[^}]*}/log: {}/g"