Knowledge Management

For HEC (httpInputQ) set maxSize in server.conf

hrawat
Splunk Employee
Splunk Employee

There is a practice of setting queueSize in inputs.conf [http://<token>] stanza. queueSize over writes server.conf stanza

 

 

[queue=httpInputQ]
maxSize

 


Now if you have multiple tokens with different queueSize.

 

 

inputs.conf
[http://1]
queueSize=1

[http://2]
queueSize=2

[http://3]
queueSize=3

[http://4]
queueSize=4

 

 

Globally only one inputs.conf stanza wins for final httpInputQ size.

This setting should only be set if setting 'persistentQueueSize' as well. If there are multiple http inputs configured and each input has set 'queueSize' but persistentQueueSize is not is set, splunkd will create one in-memory queue and pick the 'queueSize' value from first stanza after sorting http stanzas with matching token of first received http event in ascending order. With multiple pipelines configured, each pipeline will create one in-memory queue depending on the first http event received by the pipeline thus each pipeline might have different sized httpInputQ created. If there are multiple http stanzas configured and 'persistentQueueSize' is not set, prefer to set 'maxSize' under 'queue=httpInputQ' stanza in server.conf.


So best practice would be to never set per token queueSize in inputs.conf. Instead set one time in server.conf, if not setting persistentQueueSize.

 

 

[queue=httpInputQ]
maxSize

 

 

Tags (1)

hrawat
Splunk Employee
Splunk Employee

We are updating docs to reflect layering of multiple http stanzas with different queueSize values.
Eventually all tokens share one input queue httpInputQ. Once all tokens are read in-memory the first token(shorted in ascending order) wins and creates final httpInputQ. Other queueSize values are no-op since the queue is already created.

Above is also applicable for multiple splunktcpin or tcpin ports having different queueSize but sharing splunktcp queue or tcpin queue.

isoutamo
SplunkTrust
SplunkTrust

Thanks @hrawat 

I just check this from docs and nether inputs.conf, server.conf or Set up and use HTTP Event Collector with configuration files says anything that there is only one value for queueSize. At least me, as non native English speaker, cannot get that conclusion based on those documents. It's much easier to understand this just opposite way. 

Have you already asked fixes/additional information for those documents?

r. Ismo

0 Karma

PickleRick
SplunkTrust
SplunkTrust

Yes. That is how I'd interpret the inputs.conf spec as well.

I can understand though why would just one value be effective (it's after all just one input bound to one port and the data is just internally split between various tokens) but the docs are ambiguous on this one to say the least.

 

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Agent Mode Engaged! Enchaining Agentic Operations with Splunk AI Assistant 2.0

    Are you ready to transform how your team handles complex data requests? We invite you to our upcoming ...

Announcing Modern Navigation: A New Era of Splunk User Experience

We are excited to introduce the Modern Navigation feature in the Splunk Platform, available to both cloud and ...

Modernize your Splunk Apps – Introducing Python 3.13 in Splunk

We are excited to announce that the upcoming releases of Splunk Enterprise 10.2.x and Splunk Cloud Platform ...