Does an indexer cluster or indexer get any acknowledgment that the data is received ?

[anem]

Hi all,

I have this question and couldn't find the answers so far so posting here in hoping to find some knowledge.

Q1) When a universal forwarder sends logs based in inputs.conf/ http even collector to an Indexer cluster or indexer does it gets any acknowledgment that the data is received?

[PickleRick]

It depends on the outputs.conf settings.

useACK = <boolean>
* Whether or not to use indexer acknowledgment.
* Indexer acknowledgment is an optional capability on forwarders that helps
  prevent loss of data when sending data to an indexer.
* When set to "true", the forwarder retains a copy of each sent event
  until the receiving system sends an acknowledgment.
  * The receiver sends an acknowledgment when it has fully handled the event
    (typically when it has written it to disk in indexing).
  * If the forwarder does not receive an acknowledgment, it resends the data
    to an alternative receiver.
  * NOTE: The maximum memory used for the outbound data queues increases
    significantly by default (500KB -> 28MB) when the 'useACK' setting is
    enabled. This is intended for correctness and performance.
* When set to "false", the forwarder considers the data fully processed
  when it finishes writing it to the network socket.
* You can configure this setting at the [tcpout] or [tcpout:<target_group>]
  stanza levels. You cannot set it for individual servers at the
  [tcpout-server: ...] stanza level.
* Default: false

[anem]

@PickleRick hi thank you for the answer, do we have any queue's at indexer cluster or indexer or forwarder manager so in case of any performance issues and indxing is slow and data could be in queue and there would be no data loss

[PickleRick]

maxQueueSize = [<integer>|<integer>[KB|MB|GB]|auto]
* The maximum size of the forwarder output queue.
* The size can be limited based on the number of entries, or on the total
  memory used by the items in the queue.
* If specified as a lone integer (for example, "maxQueueSize=100"),
  the 'maxQueueSize' setting indicates the maximum count of queued items.
* If specified as an integer followed by KB, MB, or GB
  (for example, maxQueueSize=100MB), the 'maxQueueSize' setting indicates
  the maximum random access memory (RAM) size of all the items in the queue.
* If set to "auto", this setting configures a value for the output queue
  depending on the value of the 'useACK' setting:
  * If 'useACK' is set to "false", the output queue uses 500KB.
  * If 'useACK' is set to "true", the output queue uses 7MB.
* If you enable indexer acknowledgment by configuring the 'useACK'
  setting to "true", the forwarder creates a wait queue where it temporarily
  stores data blocks while it waits for indexers to acknowledge the receipt
  of data it previously sent.
  * The forwarder sets the wait queue size to triple the value of what
    you set for 'maxQueueSize.'
  * For example, if you set "maxQueueSize=1024KB" and "useACK=true",
    then the output queue is 1024KB and the wait queue is 3072KB.
  * Although the wait queue and the output queue sizes are both controlled
    by this setting, they are separate.
  * The wait queue only exists if 'useACK' is set to "true".
* Limiting the queue sizes by quantity is historical. However,
  if you configure queues based on quantity, keep the following in mind:
  * Queued items can be events or blocks of data.
    * Non-parsing forwarders, such as universal forwarders, send
      blocks, which can be up to 64KB.
    * Parsing forwarders, such as heavy forwarders, send events, which
      are the size of the events. Some events are as small as
      a few hundred bytes. In unusual cases (data dependent), you might
      arrange to produce events that are multiple megabytes.
* Default: auto
  * if 'useACK' is set to "true" and this setting is set to "auto", then
    the output queue is 7MB and the wait queue is 21MB.

See the output.conf docs 🙂

[SanjayReddy · Accepted solution]

Hi @anem 

Yes,We have  quques each level that helps to identify the issue if blocking at any quque 

1.parsingQueue
2.aggQueue/merginig
3.typingQueue
4.indexQueue

you can check quque performance from  Monioting Console 

Indexing->Performance->Indexing Performance : Instance.

using query 
index=_internal sourcetype=splunkd source=*metrics.log group=queue blocked=true 

also @gcusello  given good query to find out queue filledup values please refer to post tohttps://community.splunk.com/t5/Getting-Data-In/How-do-we-know-whether-typing-queues-are-blocked-or-...   

additional resuoures about queues 

https://wiki.splunk.com/Community:HowIndexingWorks#:~:text=Splunk%20processes%20data%20through%20pip.... 
 
https://conf.splunk.com/files/2019/slides/FN1570.pdf