topic Filter a log using regex in Splunk Search

Filter a log using regex

jose_sepulveda — Tue, 18 Jun 2024 14:26:04 GMT

I need to filter a part of a log using regex, I have the following log

log: {dx.trace_id=xxxxx, dx.span_id=yyyyy, dx.trace_sampled=true}{"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

I need to remove this fragment from the answer

{dx.trace_id=xxxxx, dx.span_id=yyyyy, dx.trace_sampled=true}

so that the visible log is the following

log: {"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

There are also outputs where what I need to filter is presented with fewer fields or without fields, leaving it this way

log: {dx.trace_sampled=true}{"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

log: {}{"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

In these last two examples I still need to filter the following respectively

{dx.trace_sampled=true}

{}

So that the output is finally clean and leaves only what I need

I hope you can help me please

Re: Filter a log using regex

ITWhisperer — Tue, 18 Jun 2024 14:48:09 GMT

| rex mode=sed "s/log: {[^}]*}/log: {}/g"

Re: Filter a log using regex

gcusello — Tue, 18 Jun 2024 14:48:55 GMT

Hi @jose_sepulveda,

at first: do you want to remove a part of your logs before indexing or at search time?

if at index time remember that in this way you change the format of your logs, so the add-ons could not work correctly!

anyway, why do you want to remove a part of your logs?

your request doesn't seem to be an obfuscation.

anyway, you can do this using the command SEDCMD in the props.con using a substitution regex like the following:

SEDCMD = s/([^\}]+\})(.*)/$2/g

Ciao.

Giuseppe

Re: Filter a log using regex

jose_sepulveda — Tue, 18 Jun 2024 15:06:29 GMT

Hi @gcusello

I have a service developed in JAVA that is dockerized, a shared tomcat image is used that is adding these fragments to the service's output logs, which are the ones I really need to view in splunk. For this reason the response is no longer a valid Json and the visualization is presented as a string and I need to resolve that situation

Re: Filter a log using regex

gcusello — Tue, 18 Jun 2024 15:11:21 GMT

Hi @jose_sepulveda ,

ok, please, check my solution or the one from @ITWhisperer that's similar.

Ciao.

Giuseppe

Re: Filter a log using regex

jose_sepulveda — Tue, 18 Jun 2024 15:26:36 GMT

Hi @gcusello , @ITWhisperer

When adding the regular expression indicated by @ITWhisperer to the search, it still appears in the result, which I need to remove

Re: Filter a log using regex

ITWhisperer — Tue, 18 Jun 2024 17:35:40 GMT

This looks like a corrupted / non-standard version of JSON (It would be helpful for you to share the unformatted version of the log since that is what the rex will be working with!). Try something like this

| rex mode=sed "s/\"log\": {[^}]*}/\"log\": {}/g"