Getting Data In

Modify _raw but keep extracted fields


Hi All,

Today I encapsulate system logs in a JSON structure in order to add metadata that I would like to add to Splunk:

Raw field:

      "message": 2018-05-21T04:36:52.685 WARN: This is a warning",

The actual log line is in the "message" field. Because this is JSON, Splunk parses it easily and extracts the fields nicely.

Now my issue is when I perform searches on this data, having the whole JSON structure in the _raw field is very cumbersome and prevents me from using the normal events viewer to browse the logs (because of all the extra information around the data i'm interested in)

My questions:
- Is there a way (via transforms/props) to replace _raw by the field "message" while keeping the fields saved at index time?
-If the above solution is not possible/not clean, is there a way for me to somehow send metadata on top of _raw when i send the data via TCP?

Thanks in advance,

0 Karma


If you want to keep the metadata, and only show the message in the search results you could try this in your props.conf:


EVAL-_raw = 'fields.message'

This will keep the other field information that was extracted, but only show the fields.message JSON data as the event.

0 Karma

Path Finder

I will suggest you to use " | fields - _raw" in your search to remove the raw data on the search time .

0 Karma

Path Finder

Hi Benoit,

I am also running across this problem. If you found a solution, please share!


0 Karma

Super Champion

if you want entire raw text in a message field then you can use calculated fields knowledge objects.
go to Fields » Calculated fields » Add new
Put necessary details and in eval Expression put message=_raw and save.

let me know if this helps!

0 Karma


I would be more interested in _raw=message.
Do you think it would be possible with calculated fields?

0 Karma


The extracted fields from your JSON data are not created at index-time but at search-time, so somehow changing _raw from its JSON format will also break the field extraction.

What kind of metadata are you missing in your current _raw that you'd want to include?

0 Karma


Good to know for the JSON extraction.

For the metadata, some of the fields I'm interested in are:
- The original file as source (lost as I'm sending data via TCP)
- The Site where the data is coming from (I thought about having one TCP input configured for each, but it's not very easy to maintain)

There might be extra fields.

0 Karma
Get Updates on the Splunk Community!

Splunk Security Content for Threat Detection & Response, Q1 Roundup

Join Principal Threat Researcher, Michael Haag, as he walks through:An introduction to the Splunk Threat ...

Splunk Life | Happy Pride Month!

Happy Pride Month, Splunk Community! 🌈 In the United States, as well as many countries around the ...

SplunkTrust | Where Are They Now - Michael Uschmann

The Background Five years ago, Splunk published several videos showcasing members of the SplunkTrust to share ...