topic Re: Modify _raw but keep extracted fields in Getting Data In

Modify _raw but keep extracted fields

bntdumas — Mon, 21 May 2018 04:49:18 GMT

Hi All,

Today I encapsulate system logs in a JSON structure in order to add metadata that I would like to add to Splunk:

Raw field:

 {"@metadata":
    {"type":"log","beat":"filebeat"},
    "beat":{
      "name":"server3",
      "hostname":"server3",
      "version":"5.4.0"},
    "fields":{
      "siteId":"Sweden"
      "source":"/root/log/currentLog.log"
      ,"offset":916645,
      "type":"log",
      "message": 2018-05-21T04:36:52.685 WARN: This is a warning",
      "@timestamp":"2018-05-21T02:37:27.919Z",
       "input_type":"log"
}

The actual log line is in the "message" field. Because this is JSON, Splunk parses it easily and extracts the fields nicely.

Now my issue is when I perform searches on this data, having the whole JSON structure in the _raw field is very cumbersome and prevents me from using the normal events viewer to browse the logs (because of all the extra information around the data i'm interested in)

My questions:
- Is there a way (via transforms/props) to replace _raw by the field "message" while keeping the fields saved at index time?
-If the above solution is not possible/not clean, is there a way for me to somehow send metadata on top of _raw when i send the data via TCP?

Thanks in advance,
Benoit

Re: Modify _raw but keep extracted fields

Ayn — Mon, 21 May 2018 05:11:47 GMT

The extracted fields from your JSON data are not created at index-time but at search-time, so somehow changing _raw from its JSON format will also break the field extraction.

What kind of metadata are you missing in your current _raw that you'd want to include?

Re: Modify _raw but keep extracted fields

mayurr98 — Mon, 21 May 2018 05:45:30 GMT

if you want entire raw text in a message field then you can use calculated fields knowledge objects.
go to Fields » Calculated fields » Add new
Put necessary details and in eval Expression put message=_raw and save.

let me know if this helps!

Re: Modify _raw but keep extracted fields

bntdumas — Mon, 21 May 2018 06:28:54 GMT

Good to know for the JSON extraction.

For the metadata, some of the fields I'm interested in are:
- The original file as source (lost as I'm sending data via TCP)
- The Site where the data is coming from (I thought about having one TCP input configured for each, but it's not very easy to maintain)

There might be extra fields.

Re: Modify _raw but keep extracted fields

bntdumas — Mon, 21 May 2018 06:30:52 GMT

I would be more interested in _raw=message.
Do you think it would be possible with calculated fields?

Re: Modify _raw but keep extracted fields

alanzchan — Sun, 17 Mar 2019 01:42:50 GMT

Hi Benoit,

I am also running across this problem. If you found a solution, please share!

-Alan

Re: Modify _raw but keep extracted fields

SathyaNarayanan — Mon, 18 Mar 2019 07:16:18 GMT

I will suggest you to use " | fields - _raw" in your search to remove the raw data on the search time .

Re: Modify _raw but keep extracted fields

jnudell_2 — Wed, 30 Sep 2020 01:03:37 GMT

If you want to keep the metadata, and only show the message in the search results you could try this in your props.conf:

props.conf

[my_sourcetype]

EVAL-_raw = 'fields.message'

This will keep the other field information that was extracted, but only show the fields.message JSON data as the event.