topic Re: Field extract from Rex in Splunk Search

How to do Field extraction from Rex?

Sangamesh — Mon, 21 Aug 2023 19:51:32 GMT

I need to extract the values between >>>>|| || and after the >>>>|| || referring the below sample and output should be like

values between>>>>||1407|| should be temp=1407

values after >>>>||1407|| should be message=[POD CleanUp] File deleted from POD : /dfgd/dfgdfgdfg.dat

Here is the sample log:

{"source":"fdgdfdfg","log":"2023-08-21 04:07:12.400 INFO 42 --- [dfgdf] c.j.t.f.dgf.dfgd.dgf : >>>>||1407|| [POD CleanUp] File deleted from POD : /dfgd/dfgdfgdfg.dat","host":"xx-ret353.svr.gg.fghs.net","tags":["_dateparsefailure"],"@version":"1","Kubernetes.pod":"gkp-xcs-services-black-prd-67986d784-b6c5j","s_sourcetype":"tyu","@timestamp":"2023-08-21T08:07:28.420Z","Kubernetes.namespace":"80578d64606-56-fyt-ty-prod","appId":"1235","app_id":"2345","log_file":"/app/logs/app.log","Kubernetes.node":"sd-1564sw32b0f.svr.us.sdf.net"}

@ITWhisperer

Re: Field extract from Rex

ITWhisperer — Mon, 21 Aug 2023 14:40:51 GMT

This looks like JSON, so assuming you have already extract the log field, try this

| rex field=log ">>>>\|\|(?<temp>[^\|]+)\|\|\s(?<message>.+)"

Re: Field extract from Rex

Sangamesh — Wed, 23 Aug 2023 13:18:47 GMT

How to extract these fields timestamp,kubernetes.pod too along with the below provided solutions

@ITWhisperer

Re: Field extract from Rex

isoutamo — Wed, 23 Aug 2023 13:54:51 GMT

one way to do it use separate rex expressions. Then it's not dependent on order of those values in your log message. If you could be sure that order is always same then you can add all in one or to rex. As you have json (based on your examples) you could also use extract/kv command to extract those fields like json.

... | rex "timestamp\":\"(?<timestamp>\d{4}-\d\d-\d\dT\d\d:\d\d:\d\d.\d{3}[^\"]+)" | rex "Kubernetes\.pod\":\"(?<kubernets_pod>[^\"]+)"

r. Ismo

Added missed ) for 1st one.

Re: Field extract from Rex

Sangamesh — Wed, 23 Aug 2023 13:50:25 GMT

@isoutamo @Whatever you provided solution not extracting the timestamp and Kubernetes.pod

Re: Field extract from Rex

isoutamo — Wed, 23 Aug 2023 13:57:04 GMT

Based on your example this should works after I fix/add missed ) on timestamp part.

Timestamps https://regex101.com/r/QkHusK/1
Kubernetes.pod https://regex101.com/r/vWFIMe/1

Re: Field extract from Rex

ITWhisperer — Wed, 23 Aug 2023 13:58:20 GMT

Given that this looks like JSON, you should either already have these fields if you have ingested the log correctly, or you could use spath to extract them. If you want to continue with rex for these fields, try this:

| rex "timestamp\":\"(?<timestamp>[^\"]+)" | rex "Kubernetes.pod\":\"(?<kubernetes_pod>[^\"]+)"

Re: Field extract from Rex

Sangamesh — Mon, 28 Aug 2023 12:23:23 GMT

I see couple of logs starting with this log format too <><><><>||1407||

could you please provide the Rex expression with already provided solution @ITWhisperer

Re: Field extract from Rex

Sangamesh — Tue, 29 Aug 2023 07:03:30 GMT

@ITWhisperer @Will you able to provide the Rex for the below log format too.

<><><><>||1407||

Re: Field extract from Rex

ITWhisperer — Tue, 29 Aug 2023 07:50:17 GMT

| rex field=log "\<>\<>\<>\<>\|\|(?<temp>[^\|]+)\|\|\s(?<message>.+)"

Re: Field extract from Rex

Sangamesh — Tue, 29 Aug 2023 09:36:06 GMT

@ITWhisperer @Whatever you provided rex expression is not fetching the values

Re: Field extract from Rex

ITWhisperer — Tue, 29 Aug 2023 09:40:43 GMT

Please share the complete event which is not working for you (anonymised of course). Please use a code block </> so the formatting and special characters are preserved.