Splunk Search

Filter a log using regex

jose_sepulveda
Loves-to-Learn

I need to filter a part of a log using regex, I have the following log

log: {dx.trace_id=xxxxx, dx.span_id=yyyyy, dx.trace_sampled=true}{"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

I need to remove this fragment from the answer

{dx.trace_id=xxxxx, dx.span_id=yyyyy, dx.trace_sampled=true}

so that the visible log is the following

log: {"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

There are also outputs where what I need to filter is presented with fewer fields or without fields, leaving it this way

log: {dx.trace_sampled=true}{"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

log: {}{"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

In these last two examples I still need to filter the following respectively

{dx.trace_sampled=true}

{}

So that the output is finally clean and leaves only what I need

log: {"logtopic":"x","appname":"y","module":"z","Id":"asdasd","traceId":"aaaaaaa","parentId":"sssssss","spanId":"ddddddd","traceFlags":"00","timestamp":"2024-05-29 11:42:37.675","event":"POST:geAll","level":"info","payload":{"orderId":"yyyy","channel":"zzz","skupCheck":true},"msgResponse":{"httpMethod":"POST","httpStatusCode":200,"httpMessage":"OK","url":"getAll"},"message":"Response in POST:getAll"}

I hope you can help me please

 

Labels (3)
Tags (2)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jose_sepulveda,

at first: do you want to remove a part of your logs before indexing or at search time?

if at index time remember that in this way you change the format of your logs, so the add-ons could not work correctly!

anyway, why do you want to remove a part of your logs?

your request doesn't seem to be an obfuscation.

anyway, you can do this using the command SEDCMD in the props.con using a substitution regex like the following:

SEDCMD = s/([^\}]+\})(.*)/$2/g

Ciao.

Giuseppe

0 Karma

jose_sepulveda
Loves-to-Learn

Hi @gcusello 

I have a service developed in JAVA that is dockerized, a shared tomcat image is used that is adding these fragments to the service's output logs, which are the ones I really need to view in splunk. For this reason the response is no longer a valid Json and the visualization is presented as a string and I need to resolve that situation

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @jose_sepulveda ,

ok, please, check my solution or the one from @ITWhisperer that's similar.

Ciao.

Giuseppe

0 Karma

jose_sepulveda
Loves-to-Learn

Hi @gcusello , @ITWhisperer 

When adding the regular expression indicated by @ITWhisperer  to the search, it still appears in the result, which I need to remove

Captura de pantalla 2024-06-18 a la(s) 11.18.31 a. m..png

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

This looks like a corrupted / non-standard version of JSON (It would be helpful for you to share the unformatted version of the log since that is what the rex will be working with!). Try something like this

| rex mode=sed "s/\"log\": {[^}]*}/\"log\": {}/g"
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust
| rex mode=sed "s/log: {[^}]*}/log: {}/g"
0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...