Getting Data In

Modify _raw but keep extracted fields

bntdumas
Engager

Hi All,

Today I encapsulate system logs in a JSON structure in order to add metadata that I would like to add to Splunk:

Raw field:

 {"@metadata":
    {"type":"log","beat":"filebeat"},
    "beat":{
      "name":"server3",
      "hostname":"server3",
      "version":"5.4.0"},
    "fields":{
      "siteId":"Sweden"
      "source":"/root/log/currentLog.log"
      ,"offset":916645,
      "type":"log",
      "message": 2018-05-21T04:36:52.685 WARN: This is a warning",
      "@timestamp":"2018-05-21T02:37:27.919Z",
       "input_type":"log"
}

The actual log line is in the "message" field. Because this is JSON, Splunk parses it easily and extracts the fields nicely.

Now my issue is when I perform searches on this data, having the whole JSON structure in the _raw field is very cumbersome and prevents me from using the normal events viewer to browse the logs (because of all the extra information around the data i'm interested in)

My questions:
- Is there a way (via transforms/props) to replace _raw by the field "message" while keeping the fields saved at index time?
-If the above solution is not possible/not clean, is there a way for me to somehow send metadata on top of _raw when i send the data via TCP?

Thanks in advance,
Benoit

0 Karma

jnudell_2
Builder

If you want to keep the metadata, and only show the message in the search results you could try this in your props.conf:

props.conf

[my_sourcetype]
EVAL-_raw = 'fields.message'

This will keep the other field information that was extracted, but only show the fields.message JSON data as the event.

0 Karma

SathyaNarayanan
Path Finder

I will suggest you to use " | fields - _raw" in your search to remove the raw data on the search time .

0 Karma

alanzchan
Path Finder

Hi Benoit,

I am also running across this problem. If you found a solution, please share!

-Alan

0 Karma

mayurr98
Super Champion

if you want entire raw text in a message field then you can use calculated fields knowledge objects.
go to Fields » Calculated fields » Add new
Put necessary details and in eval Expression put message=_raw and save.

let me know if this helps!

0 Karma

bntdumas
Engager

I would be more interested in _raw=message.
Do you think it would be possible with calculated fields?

0 Karma

Ayn
Legend

The extracted fields from your JSON data are not created at index-time but at search-time, so somehow changing _raw from its JSON format will also break the field extraction.

What kind of metadata are you missing in your current _raw that you'd want to include?

0 Karma

bntdumas
Engager

Good to know for the JSON extraction.

For the metadata, some of the fields I'm interested in are:
- The original file as source (lost as I'm sending data via TCP)
- The Site where the data is coming from (I thought about having one TCP input configured for each, but it's not very easy to maintain)

There might be extra fields.

0 Karma
Get Updates on the Splunk Community!

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...