KV limit in limits.conf

[manthantsarwade]

Hi,

We develop a Splunk app that ingests JSON events with 150+ fields. Our props.conf uses:

[our_sourcetype]

KV_MODE = json

TRUNCATE = 0

Customer Issue: A customer reports that fields beyond ~100 are not being extracted, causing their detection rules to fail silently. 

They believe the [kv] limit = 100 in limits.conf is causing field truncation.

For the workaround and time being they have increased it to 200.

Constraints:

• Customer cannot increase [kv] limits due to CPU/performance concerns.
• Writing regex-based extractions for 100+ nested/array fields is not maintainable.
• We cannot modify the JSON structure at the source.

Is there a supported way to ensure full JSON extraction without raising KV limits?

Is there any recommended pattern for handling complex nested JSON arrays in extraction?

Please guide me what could be the best possible solution.

Also does the 100 field limit in [kv] stanza apply to KV_MODE = json, or only to KV_MODE = auto?

[PrewinThomas]

@manthantsarwade 

The [kv] limit in limits.conf does apply to KV_MODE=json as well as KV_MODE=auto.
If you cant increase your resources, then instead of relying on auto‑KV, you can extract required fields with spath at search time.

Also you can try with INDEXED_EXTRACTIONS This extracts JSON fields at index time, bypassing the [kv] search‑time limit.

Eg:

[your_sourcetype]
INDEXED_EXTRACTIONS = json

Note: When 'INDEXED_EXTRACTIONS = JSON' for a particular source type, do not also set 'KV_MODE = json'

Regards,
Prewin
🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!

[manthantsarwade]

Hi @PrewinThomas,

Thank you for the response.😄

As you mentioned that the KV limit does not apply when KV_MODE=json, we validated this during our internal testing and observed that more than 200 fields are being extracted successfully. However, on the client side, the same fields are not being extracted consistently. We are trying to understand if there are stricter limits or additional restrictions on other Splunk platforms, such as Splunk Victoria or Splunk Cloud.

Regarding indexed_extractions, we understand that it still has a limit of 200 fields. In our case, the events are highly dynamic (containing arrays and deeply nested structures) and can reach 250+ fields in some scenarios.

The client also mentioned that their detection rules are failing because the fields are not being extracted. Since detection rules operate at search time, would it be feasible for them to use spath within their detection rules so that the required fields are extracted at search time and become available for detection logic?

Please let us know your thoughts or if there are any recommended best practices for handling this scenario in Splunk Cloud or other environments.

[PrewinThomas]

@manthantsarwade 

On the limits setting i dont think classic and victoria have much difference, other than the below highlighted ones.

#https://help.splunk.com/en/splunk-cloud-platform/administer/admin-manual/10.1.2507/get-started-manag...

For this scenario, my suggesstion would be
Avoid relying on KV_MODE=json for very large payloads. Use spath in searches or detection rules for critical fields.
Use field aliases or EVAL in props.conf for the most important fields, so they’re always available without depending on auto‑KV.
For detection rules- explicitly call spath to guarantee extraction.
Performance - extracting 250+ fields at search time can be expensive. Focus on the subset of fields that detection rules truly need.

Regards,
Prewin
🌟If this answer helped you, please consider marking it as the solution or giving a Karma. Thanks!