topic Re: Should you add the configurations in limits.conf on universal forwarder OR indexer servers? in Getting Data In

Should you add the configurations in limits.conf on universal forwarder OR indexer servers?

manjunathmeti — Wed, 13 Mar 2019 11:38:51 GMT

We have a universal forwarder that monitors json files with number of keys>500. We need to parse this during index time, since we don't want to affect search performance during search time. By default splunk only extracts 100 fields and I need to add below configurations in limits.conf to increase this.

[kv]
avg_extractor_time = 500
limit = 1000
max_extractor_time = 1000
maxchars = 51200
maxcols = 1024

My question is where do I need to add these configurations, on universal forwarder or on indexer servers?

I referred "4. Detail Diagram - UF/LWF to Indexer" in this page https://wiki.splunk.com/Community:HowIndexingWorks. But this doesn't tell where to configure inputs.conf exactly.

One more thing below configurations are already added in props.conf on universal forwarder to parse json data.

[sourcetype]
INDEXED_EXTRACTIONS = json
KV_MODE=none

Re: Should you add the configurations in limits.conf on universal forwarder OR indexer servers?

nickhills — Wed, 13 Mar 2019 11:41:12 GMT

Hi @manjunathmeti,

A UF does not do any parsing or any preprocessing, all of this happens on the indexers, so the limits.conf which applies is therefore the one on the indexers.

The same is true for props.conf in your example, though there is no problem with having that on the UF too.

Re: Should you add the configurations in limits.conf on universal forwarder OR indexer servers?

DMohn — Wed, 13 Mar 2019 11:44:48 GMT

Regarding your limits.conf - this setting is needed on the indexing tier, as structured field extraction happens in the Typing Tier of the indexing pipeline, which will always be on the indexer. This is stated in the Wiki reference you provided as well. (It doesn't mention limits.conf exactly, but field extraction is done during the typing, hence it is affected by the limits you want to set).

Re: Should you add the configurations in limits.conf on universal forwarder OR indexer servers?

manjunathmeti — Thu, 21 Mar 2019 02:59:07 GMT

Thank you DMohn.

Re: Should you add the configurations in limits.conf on universal forwarder OR indexer servers?

manjunathmeti — Thu, 21 Mar 2019 02:59:38 GMT

Thank you Nickhillscpl.