Getting Data In

Missing events? JSON payload and indexed_extractions

swangertyler
Path Finder

We have events where the JSON payload has 100s of fields. When I table a field, we can see entries for some events but not others. However, if I spath the field beforehand, I then can discover it. We are using indexed_extractions? Is there a limit with this?

1 Solution

kamlesh_vaghela
SplunkTrust
SplunkTrust

@swangertyler

As your single JSON event has 100+ fields, so extracting first 100 fields is the default behaviour of the Splunk. You can set this number by updating limits.conf.

 limit = <integer>
    * The maximum number of fields that an automatic key-value field extraction
      (auto kv) can generate at search time.
    * If search-time field extractions are disabled (KV_MODE=none in props.conf)
      then this setting determines the number of index-time fields that will be
      returned.
    * The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
      'linecount', 'splunk_server', and 'splunk_server_group' do not count against
      this limit and will always be returned.
    * Increase this setting if, for example, you have indexed data with a large
      number of columns and want to ensure that searches display all fields from
      the data.
    * Default: 100

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bkv.5D

If your JSON length is 5000+ character than you can also try by configuring extraction_cutoff of spath stanza.

extraction_cutoff = <integer>
* For extract-all spath extraction mode, only apply extraction to the first
  <integer> number of bytes.
* Default: 5000

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bspath.5D

View solution in original post

kamlesh_vaghela
SplunkTrust
SplunkTrust

@swangertyler

As your single JSON event has 100+ fields, so extracting first 100 fields is the default behaviour of the Splunk. You can set this number by updating limits.conf.

 limit = <integer>
    * The maximum number of fields that an automatic key-value field extraction
      (auto kv) can generate at search time.
    * If search-time field extractions are disabled (KV_MODE=none in props.conf)
      then this setting determines the number of index-time fields that will be
      returned.
    * The summary fields 'host', 'index', 'source', 'sourcetype', 'eventtype',
      'linecount', 'splunk_server', and 'splunk_server_group' do not count against
      this limit and will always be returned.
    * Increase this setting if, for example, you have indexed data with a large
      number of columns and want to ensure that searches display all fields from
      the data.
    * Default: 100

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bkv.5D

If your JSON length is 5000+ character than you can also try by configuring extraction_cutoff of spath stanza.

extraction_cutoff = <integer>
* For extract-all spath extraction mode, only apply extraction to the first
  <integer> number of bytes.
* Default: 5000

https://docs.splunk.com/Documentation/Splunk/latest/Admin/Limitsconf#.5Bspath.5D

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...