Getting Data In

Should you add the configurations in limits.conf on universal forwarder OR indexer servers?

manjunathmeti
Champion

We have a universal forwarder that monitors json files with number of keys>500. We need to parse this during index time, since we don't want to affect search performance during search time. By default splunk only extracts 100 fields and I need to add below configurations in limits.conf to increase this.

[kv]
avg_extractor_time = 500
limit = 1000
max_extractor_time = 1000
maxchars = 51200
maxcols = 1024

My question is where do I need to add these configurations, on universal forwarder or on indexer servers?

I referred "4. Detail Diagram - UF/LWF to Indexer" in this page https://wiki.splunk.com/Community:HowIndexingWorks. But this doesn't tell where to configure inputs.conf exactly.

One more thing below configurations are already added in props.conf on universal forwarder to parse json data.

[sourcetype]
INDEXED_EXTRACTIONS = json
KV_MODE=none
0 Karma
1 Solution

DMohn
Motivator

Regarding your limits.conf - this setting is needed on the indexing tier, as structured field extraction happens in the Typing Tier of the indexing pipeline, which will always be on the indexer. This is stated in the Wiki reference you provided as well. (It doesn't mention limits.conf exactly, but field extraction is done during the typing, hence it is affected by the limits you want to set).

View solution in original post

DMohn
Motivator

Regarding your limits.conf - this setting is needed on the indexing tier, as structured field extraction happens in the Typing Tier of the indexing pipeline, which will always be on the indexer. This is stated in the Wiki reference you provided as well. (It doesn't mention limits.conf exactly, but field extraction is done during the typing, hence it is affected by the limits you want to set).

manjunathmeti
Champion

Thank you DMohn.

0 Karma

nickhills
Ultra Champion

Hi @manjunathmeti,

A UF does not do any parsing or any preprocessing, all of this happens on the indexers, so the limits.conf which applies is therefore the one on the indexers.

The same is true for props.conf in your example, though there is no problem with having that on the UF too.

If my comment helps, please give it a thumbs up!

manjunathmeti
Champion

Thank you Nickhillscpl.

0 Karma
Get Updates on the Splunk Community!

Improve Data Pipelines Using Splunk Data Management

  Register Now   This Tech Talk will explore the pipeline management offerings Edge Processor and Ingest ...

3-2-1 Go! How Fast Can You Debug Microservices with Observability Cloud?

Register Join this Tech Talk to learn how unique features like Service Centric Views, Tag Spotlight, and ...

Thank You for Celebrating CX Day with Splunk!

Yesterday the entire team at Splunk + Cisco joined the global celebration of CX Day - celebrating our ...