Getting Data In

What are the best parameters for ingesting JSON logs?

splunkreal
Motivator

Hello,

which method is best, using TIME_PREFIX = timestamp":" or TIMESTAMP_FIELDS = @timestamp?

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/Configuretimestamprecognition#Examples does not talk about TIMESTAMP_FIELDS

We are using this parameter for another JSON source and it works fine too.

 

 

 

Examples :

UF side :
etc/deployment-apps/_server_app_LBA_ZZZ_LX/local/props.conf
[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start (useless in my opinion as should only run on indexers side?)
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z  (useless in my opinion as should only run on indexers side?)
INDEXED_EXTRACTIONS = json

etc/deployment-apps/_server_app_LBA_MIC_SUP/local/props.conf
[VVV:sup:json]
INDEXED_EXTRACTIONS = json

IDXC side :
[siem@s301lbasplmgt2 ~]$ cat /OPT/siem/splunk/etc/master-apps/APP_PROPS/local/props.conf

[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z

etc/master-apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50


SHC side:
etc/shcluster/apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
KV_MODE = none

etc/shcluster/apps/APP_YYY_parser_json/default/props.conf
[ZZZ_metrics_json]
KV_MODE = none

 

 

 

Thanks for your help.

 

* If this helps, please upvote or accept solution 🙂 *
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU..  If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

We can't validate your props.conf settings without seeing some sample data.  However, AIUI, the TIMESTAMP_FIELDS setting applies only when INDEXED_EXTRACTIONS is used so I'd use TIME_PREFIX.

---
If this reply helps you, Karma would be appreciated.

splunkreal
Motivator

It's simple JSON standard format with @timestamp field, I could upload anonymized sample data if needed 🙂

So I understand INDEXED_EXTRACTIONS = json is not necessary if I use TIME_PREFIX and I could maybe save disk space/performances not indexing fields at indexing phase (as read at https://community.splunk.com/t5/Splunk-Search/What-are-the-possible-gains-from-an-index-time-extract...)?

Thanks for your help.

 

* If this helps, please upvote or accept solution 🙂 *
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU..  If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.

---
If this reply helps you, Karma would be appreciated.

splunkreal
Motivator

Hi Rich,

that worked however I had to add SHOULD_LINEMERGE = false in indexers props.conf because our have several events at the same millisecond.

To sum up, on UF side no props.conf

On IDXC :

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50
SHOULD_LINEMERGE = false

on SHC  (app with global context) :

KV_MODE = json

 

 

 

 

* If this helps, please upvote or accept solution 🙂 *
Get Updates on the Splunk Community!

Splunk Observability Cloud | Unified Identity - Now Available for Existing Splunk ...

Raise your hand if you’ve already forgotten your username or password when logging into an account. (We can’t ...

Index This | How many sides does a circle have?

February 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

Registration for Splunk University is Now Open!

Are you ready for an adventure in learning?   Brace yourselves because Splunk University is back, and it's ...