Hello,
which method is best, using TIME_PREFIX = timestamp":" or TIMESTAMP_FIELDS = @timestamp?
https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/Configuretimestamprecognition#Examples does not talk about TIMESTAMP_FIELDS
We are using this parameter for another JSON source and it works fine too.
Examples :
UF side :
etc/deployment-apps/_server_app_LBA_ZZZ_LX/local/props.conf
[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start (useless in my opinion as should only run on indexers side?)
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z (useless in my opinion as should only run on indexers side?)
INDEXED_EXTRACTIONS = json
etc/deployment-apps/_server_app_LBA_MIC_SUP/local/props.conf
[VVV:sup:json]
INDEXED_EXTRACTIONS = json
IDXC side :
[siem@s301lbasplmgt2 ~]$ cat /OPT/siem/splunk/etc/master-apps/APP_PROPS/local/props.conf
[ZZZ_metrics_json]
TIMESTAMP_FIELDS = start
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
etc/master-apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50
SHC side:
etc/shcluster/apps/XXX_VVV_PROPS/default/props.conf
[VVV:sup:json]
KV_MODE = none
etc/shcluster/apps/APP_YYY_parser_json/default/props.conf
[ZZZ_metrics_json]
KV_MODE = none
Thanks for your help.
Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU.. If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.
We can't validate your props.conf settings without seeing some sample data. However, AIUI, the TIMESTAMP_FIELDS setting applies only when INDEXED_EXTRACTIONS is used so I'd use TIME_PREFIX.
It's simple JSON standard format with @timestamp field, I could upload anonymized sample data if needed 🙂
So I understand INDEXED_EXTRACTIONS = json is not necessary if I use TIME_PREFIX and I could maybe save disk space/performances not indexing fields at indexing phase (as read at https://community.splunk.com/t5/Splunk-Search/What-are-the-possible-gains-from-an-index-time-extract...)?
Thanks for your help.
Yes, INDEXED_EXTRACTIONS will use more storage as well as index-time CPU.. If you don't need every field from the JSON indexed then either use KV_MODE=json or extract the desired fields using EXTRACT or SPL commands.
Hi Rich,
that worked however I had to add SHOULD_LINEMERGE = false in indexers props.conf because our have several events at the same millisecond.
To sum up, on UF side no props.conf
On IDXC :
TIME_FORMAT = %Y-%m-%dT%H:%M:%S.%3N.%z
TIME_PREFIX = timestamp":"
MAX_TIMESTAMP_LOOKAHEAD = 50
SHOULD_LINEMERGE = false
on SHC (app with global context) :
KV_MODE = json