topic Re: Best parameters for ingesting JSON logs in Getting Data In

What are the best parameters for ingesting JSON logs?

splunkreal — Tue, 24 Jan 2023 15:13:20 GMT

Hello,

which method is best, using TIME_PREFIX = timestamp":" or TIMESTAMP_FIELDS = @timestamp?

https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/Configuretimestamprecognition#Examples does not talk about TIMESTAMP_FIELDS

We are using this parameter for another JSON source and it works fine too.

Examples : UF side : etc/deployment-apps/_server_app_LBA_ZZZ_LX/local/props.conf [ZZZ_metrics_json] TIMESTAMP_FIELDS = start (useless in my opinion as should only run on indexers side?) TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z (useless in my opinion as should only run on indexers side?) INDEXED_EXTRACTIONS = json etc/deployment-apps/_server_app_LBA_MIC_SUP/local/props.conf [VVV:sup:json] INDEXED_EXTRACTIONS = json IDXC side : [siem@s301lbasplmgt2 ~]$ cat /OPT/siem/splunk/etc/master-apps/APP_PROPS/local/props.conf [ZZZ_metrics_json] TIMESTAMP_FIELDS = start TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z etc/master-apps/XXX_VVV_PROPS/default/props.conf [VVV:sup:json] TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z TIME_PREFIX = timestamp":" MAX_TIMESTAMP_LOOKAHEAD = 50 SHC side: etc/shcluster/apps/XXX_VVV_PROPS/default/props.conf [VVV:sup:json] KV_MODE = none etc/shcluster/apps/APP_YYY_parser_json/default/props.conf [ZZZ_metrics_json] KV_MODE = none

Thanks for your help.

Re: Best parameters for ingesting JSON logs

richgalloway — Tue, 24 Jan 2023 14:00:04 GMT

We can't validate your props.conf settings without seeing some sample data. However, AIUI, the TIMESTAMP_FIELDS setting applies only when INDEXED_EXTRACTIONS is used so I'd use TIME_PREFIX.

Re: Best parameters for ingesting JSON logs

splunkreal — Tue, 24 Jan 2023 17:52:22 GMT

It's simple JSON standard format with @timestamp field, I could upload anonymized sample data if needed 🙂

So I understand INDEXED_EXTRACTIONS = json is not necessary if I use TIME_PREFIX and I could maybe save disk space/performances not indexing fields at indexing phase (as read at https://community.splunk.com/t5/Splunk-Search/What-are-the-possible-gains-from-an-index-time-extraction-of-a/m-p/212017)?

Thanks for your help.

Re: Best parameters for ingesting JSON logs

richgalloway — Tue, 24 Jan 2023 18:40:38 GMT

Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU.. If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.

Re: Best parameters for ingesting JSON logs

splunkreal — Wed, 25 Jan 2023 10:18:35 GMT

Hi Rich,

that worked however I had to add SHOULD_LINEMERGE = false in indexers props.conf because our have several events at the same millisecond.

To sum up, on UF side no props.conf

On IDXC :

TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50
SHOULD_LINEMERGE = false

on SHC (app with global context) :

KV_MODE = json