topic Re: No persistentqueue attributes in outputs.conf. How to configure data buffering on Heavy Forwarder ? in Getting Data In

No persistentqueue attributes in outputs.conf. How to configure data buffering on Heavy Forwarder ?

damode — Tue, 29 Sep 2020 17:03:24 GMT

I was referring to this link, https://wiki.splunk.com/Community:Best_Practice_For_Configuring_Syslog_Input to configure data buffering on Heavy Forwarder. The instructions provided talk about configuring the below attributes in outputs.conf,

maxQueueSize = integer
usePersistentQueue = true/false
persistentQueuePath = absolute_path_that_must_exist
maxPersistentQueueSizeInMegs = integer

However, except maxQueueSize, no other attributes can be found in the outputs.conf file.

Please advise how can I configure data buffering in this case.

Re: No persistentqueue attributes in outputs.conf. How to configure data buffering on Heavy Forwarder ?

woodcock — Mon, 04 Dec 2017 02:19:03 GMT

1 is in outputs.conf and 2 and 4 are deprecated. Unfortunately, the community wiki is pretty out of date and not really supported. For example, it suggests Heavy Forwarder which is actually VERY wrong in the latest versions (should always be UF):

https://www.splunk.com/blog/2016/12/12/universal-or-heavy-that-is-the-question.html

Re: No persistentqueue attributes in outputs.conf. How to configure data buffering on Heavy Forwarder ?

damode — Wed, 06 Dec 2017 23:34:54 GMT

So you mean UF can be used for data buffering ? if yes, can you please share the documentation for the configuration ? Thanks

Re: No persistentqueue attributes in outputs.conf. How to configure data buffering on Heavy Forwarder ?

woodcock — Tue, 12 Dec 2017 13:39:03 GMT

No, HF is supported but only recommended if:
1: You have to transform most of the data before sending it to indexers
OR
2: You are using Modular Inputs

Otherwise, use UF as the aggregation.

Re: No persistentqueue attributes in outputs.conf. How to configure data buffering on Heavy Forwarder ?

damode — Tue, 12 Dec 2017 21:58:05 GMT

Hi @woodcock, sorry if my question isn't clear enough.

What my main question is , how I can configure Heavy Forwarder to buffer data if Indexer goes down ?

You have mentioned that the 1, 2 and 4 settings are deprecated, but HF still supports data buffering.

So, can you please suggest what are the other steps required to configure that ?

Re: No persistentqueue attributes in outputs.conf. How to configure data buffering on Heavy Forwarder ?

woodcock — Tue, 29 Sep 2020 17:13:48 GMT

Your best bet is to read these 4 files:

$SPLUNK_HOME/etc/system/README/outputs.conf.spec
$SPLUNK_HOME/etc/system/README/outputs.conf.examples
$SPLUNK_HOME/etc/system/README/limits.conf.spec
$SPLUNK_HOME/etc/system/README/limits.conf.examples

Re: No persistentqueue attributes in outputs.conf. How to configure data buffering on Heavy Forwarder ?

damode — Wed, 13 Dec 2017 07:23:36 GMT

found the solution at http://docs.splunk.com/Documentation/Splunk/latest/Forwarding/Protectagainstlossofin-flightdata