Splunk Search

How to extract fields from my string logs

kotig
Path Finder

I have something like below logged in as a message.
How can i replace "This is my logfile ** ->" with empty and then how to extract name, startdate, dept, enddate, status, id and get the values.
This is my logfile ** ->
myfulljson {
name {
value: "Test"
}
startdate {
value: "2020-02-21"
}
dept {
value: 110
}
enddate {
value: "20200220"
}
status {
value: "finish"
}
id {
value: "1234"
}
}

0 Karma

woodcock
Esteemed Legend

Your trash log is not even CLOSE to valid JSON.
I used this tool to figure out how to fix it:
https://jsonlint.com/
Then I ended up with this:

| makeresults 
| eval _raw="This is my logfile ->
 myfulljson {
 name {
 value: \"Test\"
     }
 startdate {
 value: \"2020-02-21\"
     }
 dept {
 value: 110
 }
 enddate {
 value: \"20200220\"
     }
 status {
 value: \"finish\"
     }
 id {
 value: \"1234\"
     }
 }"
| rename COMMENT AS "Strip off the header garbage"
| rex mode=sed "s/^This is my logfile\s*\->[\r\n\s]*myfulljson[\r\n\s]*//"
| rename COMMENT AS "Transform the non-JSON to mostly-compliant-JSON"
| rex mode=sed "s/(\w+)([\r\n\s]*\{)/\"\1\":\2/g s/value: /\"value\": /g s/\}([\r\n\s]*\")/},\1/g"
| rename COMMENT AS "OKish JSON is now parseable with 'KV_MODE=json'"
| kv

woodcock
Esteemed Legend

To do this properly, you should transform this garbage into fully-compliant JSON (that means removing ALL whitesapce) BEFORE it is indexed. This will save you license and allow KV_MODE=json to "just work".

0 Karma

to4kawa
Ultra Champion

UPDATE:

| makeresults 
| eval _raw="This is my logfile ->
myfulljson {
name {
value: \"Test\"
}
startdate {
value: \"2020-02-21\"
}
dept {
value: 110
}
enddate {
value: \"20200220\"
}
status {
value: \"finish\"
}
id {
value: \"1234\"
}
}" 
| rex max_match=0 "(?m)(?<fieldname>[a-z]+)\s{\s*value:\s(?<fieldvalue>\S+)" 
| eval fieldvalue=trim(fieldvalue,"\"\"") 
| eval _raw=mvzip(fieldname,fieldvalue,"=") 
| kv

This is not valid JSON. spath is not useful. but, we can create valid JSON. As @manjunathmeti did.
I use rex to extract fields and values. and, rename to _raw .
so kv is useful. AS @woodcock did.

0 Karma

kotig
Path Finder

Yea tried this one unfortunately i am getting empty result. Thank you for help..

0 Karma

manjunathmeti
Champion

Try this:

<your search> | rex field=_raw mode=sed "s/This is my logfile ->\smyfulljson\s{\s/{\"/g" | rex field=_raw mode=sed "s/\s{\svalue:/\":/g" | rex field=_raw mode=sed "s/\s}\s/,\"/g" | spath

richgalloway
SplunkTrust
SplunkTrust

Just a guess.

... | eval _raw = replace ( _raw, "This is my logfile ->", "") | spath | ...
---
If this reply helps you, Karma would be appreciated.
0 Karma
Get Updates on the Splunk Community!

What the End of Support for Splunk Add-on Builder Means for You

Hello Splunk Community! We want to share an important update regarding the future of the Splunk Add-on Builder ...

Solve, Learn, Repeat: New Puzzle Channel Now Live

Welcome to the Splunk Puzzle PlaygroundIf you are anything like me, you love to solve problems, and what ...

Building Reliable Asset and Identity Frameworks in Splunk ES

 Accurate asset and identity resolution is the backbone of security operations. Without it, alerts are ...