Getting Data In

What are the best parameters for ingesting JSON logs?

splunkreal
Motivator

Hello,

which method is best, using TIME_PREFIX = timestamp":" or TIMESTAMP_FIELDS = @timestamp?

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/Configuretimestamprecognition#Examples does not talk about TIMESTAMP_FIELDS

We are using this parameter for another JSON source and it works fine too.

 

 

 

Examples :

UF side :
etc/deployment-apps/_server_app_LBA_ZZZ_LX/local/props.conf
[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start (useless in my opinion as should only run on indexers side?)
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z  (useless in my opinion as should only run on indexers side?)
INDEXED_EXTRACTIONS = json

etc/deployment-apps/_server_app_LBA_MIC_SUP/local/props.conf
[VVV:sup:json]
INDEXED_EXTRACTIONS = json

IDXC side :
[siem@s301lbasplmgt2 ~]$ cat /OPT/siem/splunk/etc/master-apps/APP_PROPS/local/props.conf

[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z

etc/master-apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50


SHC side:
etc/shcluster/apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
KV_MODE = none

etc/shcluster/apps/APP_YYY_parser_json/default/props.conf
[ZZZ_metrics_json]
KV_MODE = none

 

 

 

Thanks for your help.

 

* If this helps, please upvote or accept solution if it solved *
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU..  If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

We can't validate your props.conf settings without seeing some sample data.  However, AIUI, the TIMESTAMP_FIELDS setting applies only when INDEXED_EXTRACTIONS is used so I'd use TIME_PREFIX.

---
If this reply helps you, Karma would be appreciated.

splunkreal
Motivator

It's simple JSON standard format with @timestamp field, I could upload anonymized sample data if needed 🙂

So I understand INDEXED_EXTRACTIONS = json is not necessary if I use TIME_PREFIX and I could maybe save disk space/performances not indexing fields at indexing phase (as read at https://community.splunk.com/t5/Splunk-Search/What-are-the-possible-gains-from-an-index-time-extract...)?

Thanks for your help.

 

* If this helps, please upvote or accept solution if it solved *
0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU..  If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.

---
If this reply helps you, Karma would be appreciated.

splunkreal
Motivator

Hi Rich,

that worked however I had to add SHOULD_LINEMERGE = false in indexers props.conf because our have several events at the same millisecond.

To sum up, on UF side no props.conf

On IDXC :

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50
SHOULD_LINEMERGE = false

on SHC  (app with global context) :

KV_MODE = json

 

 

 

 

* If this helps, please upvote or accept solution if it solved *
Get Updates on the Splunk Community!

Detecting Remote Code Executions With the Splunk Threat Research Team

WATCH NOWRemote code execution (RCE) vulnerabilities pose a significant risk to organizations. If exploited, ...

Enter the Splunk Community Dashboard Challenge for Your Chance to Win!

The Splunk Community Dashboard Challenge is underway! This is your chance to showcase your skills in creating ...

.conf24 | Session Scheduler is Live!!

.conf24 is happening June 11 - 14 in Las Vegas, and we are thrilled to announce that the conference catalog ...