Splunk Search

How to extract fields from my string logs

kotig
Path Finder

I have something like below logged in as a message.
How can i replace "This is my logfile ** ->" with empty and then how to extract name, startdate, dept, enddate, status, id and get the values.
This is my logfile ** ->
myfulljson {
name {
value: "Test"
}
startdate {
value: "2020-02-21"
}
dept {
value: 110
}
enddate {
value: "20200220"
}
status {
value: "finish"
}
id {
value: "1234"
}
}

0 Karma

woodcock
Esteemed Legend

Your trash log is not even CLOSE to valid JSON.
I used this tool to figure out how to fix it:
https://jsonlint.com/
Then I ended up with this:

| makeresults 
| eval _raw="This is my logfile ->
 myfulljson {
 name {
 value: \"Test\"
     }
 startdate {
 value: \"2020-02-21\"
     }
 dept {
 value: 110
 }
 enddate {
 value: \"20200220\"
     }
 status {
 value: \"finish\"
     }
 id {
 value: \"1234\"
     }
 }"
| rename COMMENT AS "Strip off the header garbage"
| rex mode=sed "s/^This is my logfile\s*\->[\r\n\s]*myfulljson[\r\n\s]*//"
| rename COMMENT AS "Transform the non-JSON to mostly-compliant-JSON"
| rex mode=sed "s/(\w+)([\r\n\s]*\{)/\"\1\":\2/g s/value: /\"value\": /g s/\}([\r\n\s]*\")/},\1/g"
| rename COMMENT AS "OKish JSON is now parseable with 'KV_MODE=json'"
| kv

woodcock
Esteemed Legend

To do this properly, you should transform this garbage into fully-compliant JSON (that means removing ALL whitesapce) BEFORE it is indexed. This will save you license and allow KV_MODE=json to "just work".

0 Karma

to4kawa
Ultra Champion

UPDATE:

| makeresults 
| eval _raw="This is my logfile ->
myfulljson {
name {
value: \"Test\"
}
startdate {
value: \"2020-02-21\"
}
dept {
value: 110
}
enddate {
value: \"20200220\"
}
status {
value: \"finish\"
}
id {
value: \"1234\"
}
}" 
| rex max_match=0 "(?m)(?<fieldname>[a-z]+)\s{\s*value:\s(?<fieldvalue>\S+)" 
| eval fieldvalue=trim(fieldvalue,"\"\"") 
| eval _raw=mvzip(fieldname,fieldvalue,"=") 
| kv

This is not valid JSON. spath is not useful. but, we can create valid JSON. As @manjunathmeti did.
I use rex to extract fields and values. and, rename to _raw .
so kv is useful. AS @woodcock did.

0 Karma

kotig
Path Finder

Yea tried this one unfortunately i am getting empty result. Thank you for help..

0 Karma

manjunathmeti
Champion

Try this:

<your search> | rex field=_raw mode=sed "s/This is my logfile ->\smyfulljson\s{\s/{\"/g" | rex field=_raw mode=sed "s/\s{\svalue:/\":/g" | rex field=_raw mode=sed "s/\s}\s/,\"/g" | spath

richgalloway
SplunkTrust
SplunkTrust

Just a guess.

... | eval _raw = replace ( _raw, "This is my logfile ->", "") | spath | ...
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...