Getting Data In

What are the best parameters for ingesting JSON logs?

splunkreal
Motivator

Hello,

which method is best, using TIME_PREFIX = timestamp":" or TIMESTAMP_FIELDS = @timestamp?

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/Configuretimestamprecognition#Examples does not talk about TIMESTAMP_FIELDS

We are using this parameter for another JSON source and it works fine too.

 

 

 

Examples :

UF side :
etc/deployment-apps/_server_app_LBA_ZZZ_LX/local/props.conf
[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start (useless in my opinion as should only run on indexers side?)
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z  (useless in my opinion as should only run on indexers side?)
INDEXED_EXTRACTIONS = json

etc/deployment-apps/_server_app_LBA_MIC_SUP/local/props.conf
[VVV:sup:json]
INDEXED_EXTRACTIONS = json

IDXC side :
[siem@s301lbasplmgt2 ~]$ cat /OPT/siem/splunk/etc/master-apps/APP_PROPS/local/props.conf

[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z

etc/master-apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50


SHC side:
etc/shcluster/apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
KV_MODE = none

etc/shcluster/apps/APP_YYY_parser_json/default/props.conf
[ZZZ_metrics_json]
KV_MODE = none

 

 

 

Thanks for your help.

 

* If this helps, please upvote or accept solution 🙂 *
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU..  If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

We can't validate your props.conf settings without seeing some sample data.  However, AIUI, the TIMESTAMP_FIELDS setting applies only when INDEXED_EXTRACTIONS is used so I'd use TIME_PREFIX.

---
If this reply helps you, Karma would be appreciated.

splunkreal
Motivator

It's simple JSON standard format with @timestamp field, I could upload anonymized sample data if needed 🙂

So I understand INDEXED_EXTRACTIONS = json is not necessary if I use TIME_PREFIX and I could maybe save disk space/performances not indexing fields at indexing phase (as read at https://community.splunk.com/t5/Splunk-Search/What-are-the-possible-gains-from-an-index-time-extract...)?

Thanks for your help.

 

* If this helps, please upvote or accept solution 🙂 *
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU..  If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.

---
If this reply helps you, Karma would be appreciated.

splunkreal
Motivator

Hi Rich,

that worked however I had to add SHOULD_LINEMERGE = false in indexers props.conf because our have several events at the same millisecond.

To sum up, on UF side no props.conf

On IDXC :

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50
SHOULD_LINEMERGE = false

on SHC  (app with global context) :

KV_MODE = json

 

 

 

 

* If this helps, please upvote or accept solution 🙂 *
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...