Getting Data In

Missing events? JSON payload and indexed_extractions

swangertyler
Path Finder

We have events where the JSON payload has 100s of fields. When I table a field, we can see entries for some events but not others. However, if I spath the field beforehand, I then can discover it. We are using indexed_extractions? Is there a limit with this?

1 Solution

kamlesh_vaghela
SplunkTrust
SplunkTrust

@swangertyler

As your single JSON event has 100+ fields, so extracting first 100 fields is the default behaviour of the Splunk. You can set this number by updating limits.conf.

 limit = <integer>
    * The maximum number of fields that an automatic key-value field extraction
      (auto kv) can generate at search time.
    * If search-time field extractions are disabled (KV_MODE=none in props.conf)
      then this setting determines the number of index-time fields that will be
      returned.
    * The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
      'linecount', 'splunk_server', and 'splunk_server_group' do not count against
      this limit and will always be returned.
    * Increase this setting if, for example, you have indexed data with a large
      number of columns and want to ensure that searches display all fields from
      the data.
    * Default: 100

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bkv.5D

If your JSON length is 5000+ character than you can also try by configuring extraction_cutoff of spath stanza.

extraction_cutoff = <integer>
* For extract-all spath extraction mode, only apply extraction to the first
  <integer> number of bytes.
* Default: 5000

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bspath.5D

View solution in original post

kamlesh_vaghela
SplunkTrust
SplunkTrust

@swangertyler

As your single JSON event has 100+ fields, so extracting first 100 fields is the default behaviour of the Splunk. You can set this number by updating limits.conf.

 limit = <integer>
    * The maximum number of fields that an automatic key-value field extraction
      (auto kv) can generate at search time.
    * If search-time field extractions are disabled (KV_MODE=none in props.conf)
      then this setting determines the number of index-time fields that will be
      returned.
    * The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
      'linecount', 'splunk_server', and 'splunk_server_group' do not count against
      this limit and will always be returned.
    * Increase this setting if, for example, you have indexed data with a large
      number of columns and want to ensure that searches display all fields from
      the data.
    * Default: 100

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bkv.5D

If your JSON length is 5000+ character than you can also try by configuring extraction_cutoff of spath stanza.

extraction_cutoff = <integer>
* For extract-all spath extraction mode, only apply extraction to the first
  <integer> number of bytes.
* Default: 5000

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bspath.5D

Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) | New Releases

In September, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...

New in Observability - Improvements to Custom Metrics SLOs, Log Observer Connect & ...

The latest enhancements to the Splunk observability portfolio deliver improved SLO management accuracy, better ...

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...