Deployment Architecture

How to forward sub-json object to splunk indexer

jeffreysun
New Member

My application writes log data to disk file. The log data is one-line json as below. I use the splunker-forwarder to send the log to splunk indexer

{"line":{"level": "info","message": "data is correct","timestamp": "2017-08-01T11:35:30.375Z"},"source": "std"}

I want to only send the sub-json object {"level": "info","message": "data is correct","timestamp": "2017-08-01T11:35:30.375Z"} to splunk indexer, not the whole json. How should I configure splunk forwarder or splunk indexer? Is there something like the codec of logstash?

0 Karma

jkat54
SplunkTrust
SplunkTrust

Create a props.conf and put the sourcetype you are using in there like this:

[yourSourcetype]
SEDCMD-aaa=s/\{\"line\"://g     
SEDCMD-zzz=s/\,\"source\":\s\"std\"\}//g.

The props.conf settings will use perl based sed on the _raw to replace the stuff you're looking to drop with "nothing" (effectively removing it).

0 Karma

s2_splunk
Splunk Employee
Splunk Employee

To add, I have to ask: Can you change the application to remove the JSON attributes that are obviously not meaningful to you from the log message?
Just wondering, but that would obviously be the preferred solution since you are not wasting processor cycles on the Splunk side by removing it after the fact, which can be significant at high log volumes.

0 Karma

jeffreysun
New Member

Do you mean SEDCMD defined in the "props.conf" can only work in the heavy forwarder rather than the universal forwarder?

0 Karma

jkat54
SplunkTrust
SplunkTrust

SEDCMD should be used where Splunk "first sees" the data. I typically put it on the forwarders (heavy, or universal) and also on the indexers. In fact most people put the same props everywhere, indexer, forwarder, search head, etc. just because Splunk will know what to apply at search versus what to apply at index time. Plus some users like to upload sample data via the ui and you'll need the props on whatever server they upload to.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...