Modify _raw but keep extracted fields

[bntdumas]

Hi All,

Today I encapsulate system logs in a JSON structure in order to add metadata that I would like to add to Splunk:

Raw field:

 {"@metadata":
    {"type":"log","beat":"filebeat"},
    "beat":{
      "name":"server3",
      "hostname":"server3",
      "version":"5.4.0"},
    "fields":{
      "siteId":"Sweden"
      "source":"/root/log/currentLog.log"
      ,"offset":916645,
      "type":"log",
      "message": 2018-05-21T04:36:52.685 WARN: This is a warning",
      "@timestamp":"2018-05-21T02:37:27.919Z",
       "input_type":"log"
}

The actual log line is in the "message" field. Because this is JSON, Splunk parses it easily and extracts the fields nicely.

Now my issue is when I perform searches on this data, having the whole JSON structure in the _raw field is very cumbersome and prevents me from using the normal events viewer to browse the logs (because of all the extra information around the data i'm interested in)

My questions:
- Is there a way (via transforms/props) to replace _raw by the field "message" while keeping the fields saved at index time?
-If the above solution is not possible/not clean, is there a way for me to somehow send metadata on top of _raw when i send the data via TCP?

Thanks in advance,
Benoit

[jnudell_2]

If you want to keep the metadata, and only show the message in the search results you could try this in your props.conf:

props.conf

[my_sourcetype]

EVAL-_raw = 'fields.message'

This will keep the other field information that was extracted, but only show the fields.message JSON data as the event.

[SathyaNarayanan]

I will suggest you to use " | fields - _raw" in your search to remove the raw data on the search time .

[alanzchan]

Hi Benoit,

I am also running across this problem. If you found a solution, please share!

-Alan

[mayurr98]

if you want entire raw text in a message field then you can use calculated fields knowledge objects.
go to Fields » Calculated fields » Add new
Put necessary details and in eval Expression put message=_raw and save.

let me know if this helps!

[bntdumas]

I would be more interested in _raw=message.
Do you think it would be possible with calculated fields?

[Ayn]

The extracted fields from your JSON data are not created at index-time but at search-time, so somehow changing _raw from its JSON format will also break the field extraction.

What kind of metadata are you missing in your current _raw that you'd want to include?

[bntdumas]

Good to know for the JSON extraction.

For the metadata, some of the fields I'm interested in are:
- The original file as source (lost as I'm sending data via TCP)
- The Site where the data is coming from (I thought about having one TCP input configured for each, but it's not very easy to maintain)

There might be extra fields.