Getting Data In

What are the best parameters for ingesting JSON logs?

realsplunk
Motivator

Hello,

which method is best, using TIME_PREFIX = timestamp":" or TIMESTAMP_FIELDS = @timestamp?

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/Configuretimestamprecognition#Examples does not talk about TIMESTAMP_FIELDS

We are using this parameter for another JSON source and it works fine too.

 

 

 

Examples :

UF side :
etc/deployment-apps/_server_app_LBA_ZZZ_LX/local/props.conf
[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start (useless in my opinion as should only run on indexers side?)
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z  (useless in my opinion as should only run on indexers side?)
INDEXED_EXTRACTIONS = json

etc/deployment-apps/_server_app_LBA_MIC_SUP/local/props.conf
[VVV:sup:json]
INDEXED_EXTRACTIONS = json

IDXC side :
[siem@s301lbasplmgt2 ~]$ cat /OPT/siem/splunk/etc/master-apps/APP_PROPS/local/props.conf

[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z

etc/master-apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50


SHC side:
etc/shcluster/apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
KV_MODE = none

etc/shcluster/apps/APP_YYY_parser_json/default/props.conf
[ZZZ_metrics_json]
KV_MODE = none

 

 

 

Thanks for your help.

 

0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU..  If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

We can't validate your props.conf settings without seeing some sample data.  However, AIUI, the TIMESTAMP_FIELDS setting applies only when INDEXED_EXTRACTIONS is used so I'd use TIME_PREFIX.

---
If this reply helps you, Karma would be appreciated.

realsplunk
Motivator

It's simple JSON standard format with @timestamp field, I could upload anonymized sample data if needed 🙂

So I understand INDEXED_EXTRACTIONS = json is not necessary if I use TIME_PREFIX and I could maybe save disk space/performances not indexing fields at indexing phase (as read at https://community.splunk.com/t5/Splunk-Search/What-are-the-possible-gains-from-an-index-time-extract...)?

Thanks for your help.

 

0 Karma

richgalloway
SplunkTrust
SplunkTrust

Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU..  If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.

---
If this reply helps you, Karma would be appreciated.

realsplunk
Motivator

Hi Rich,

that worked however I had to add SHOULD_LINEMERGE = false in indexers props.conf because our have several events at the same millisecond.

To sum up, on UF side no props.conf

On IDXC :

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50
SHOULD_LINEMERGE = false

on SHC  (app with global context) :

KV_MODE = json

 

 

 

 

Get Updates on the Splunk Community!

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...

Observability Highlights | January 2023 Newsletter

 January 2023New Product Releases Splunk Network Explorer for Infrastructure MonitoringSplunk unveils Network ...

Security Highlights | January 2023 Newsletter

January 2023 Splunk Security Essentials (SSE) 3.7.0 ReleaseThe free Splunk Security Essentials (SSE) 3.7.0 app ...