Getting Data In

What are the requirements for a perfect Splunk JSON document?

ddrillic
Ultra Champion

Our Sales Engineer told us that the Splunk json parser requires several specific things in the json document, in order to be interpreted as json. What are they?

We would like to avoid hard-coded solutions such as How do we assign each JSON document to a distinct event?

Tags (2)
1 Solution

skoelpin
SplunkTrust
SplunkTrust

Why not just apply base configs to your JSON file and have it break correctly rather than trying to format the log to Splunk?

If you let Splunk try to figure out the linebreaking, it will add additional overhead to your indexing and slow it down.

Adding this will give you correct linebreaking and timestamping along with avoiding the merging pipeline which increases your indexing speed

[sourcetype]
TIME_PREFIX = 
TIME_FORMAT = 
LINE_BREAKER = 
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 
TRUNCATE = 

https://wiki.splunk.com/Community:HowIndexingWorks

View solution in original post

skoelpin
SplunkTrust
SplunkTrust

Why not just apply base configs to your JSON file and have it break correctly rather than trying to format the log to Splunk?

If you let Splunk try to figure out the linebreaking, it will add additional overhead to your indexing and slow it down.

Adding this will give you correct linebreaking and timestamping along with avoiding the merging pipeline which increases your indexing speed

[sourcetype]
TIME_PREFIX = 
TIME_FORMAT = 
LINE_BREAKER = 
SHOULD_LINEMERGE = false
MAX_TIMESTAMP_LOOKAHEAD = 
TRUNCATE = 

https://wiki.splunk.com/Community:HowIndexingWorks

ddrillic
Ultra Champion

@skoelpin, good question.

We have teams that can form their json logs per the Splunk's needs. So, we are lucky in this sense.

0 Karma

ddrillic
Ultra Champion

We were told by the Sales Engineer that as long as it's proper JSON, all we need to do is set -

INDEXED_EXTRACTIONS = json
category = Structured

in props.conf.

0 Karma

ddrillic
Ultra Champion

For the record, the predefined _json sourcetype has these two defined config variables -

 INDEXED_EXTRACTIONS = json
 category = Structured
0 Karma

ddrillic
Ultra Champion

This solution works!!!

skoelpin
SplunkTrust
SplunkTrust

Your sales engineer is partially right, but you should ALWAYS apply base configs to lessen the indexer load when indexing data. This is a big part of the SCC2 bootcamp

ddrillic
Ultra Champion

Much appreciated @skoelpin.

Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...