All Apps and Add-ons

When HF with "Splunk DB Connect" send data to Indexer, and if Indexer is down, the data will be lost?

yutaka1005
Builder

If there is a environment like below, if Indexer is down, the data will be lost?

*HF with Splunk DB Connect
*Indexer

I think data input like "file monitor","data from other Universal Forwarder" etc will stop, if indexer is down, so the data won't be lost.

But I have to configure persistent queue to avoid data lost, if I use data input such "TCP","scripted input".

which type is data input of Splunk DB Connet?
Do I have to configure persistent queue?
Or Splunk DB Connet automatically stop, if outputqueue is filled?

If someone know about it , please tell me.

0 Karma

harsmarvania57
Ultra Champion

Splunk DB Connect 3 is using HEC (HTTP Event Collector) to ingest data and based on how indexing works diagram https://wiki.splunk.com/Community:HowIndexingWorks, HEC uses httpinputq but I am not able to find anything related to httpinpuq in Splunk Docs and not sure how DB connect on HFW will react when Indexer is down.

EDIT: However based on https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Inputsconf#HTTP_Event_Collector_.28HEC.29_-... , you can set persistentQueueSize for [http://<token>]

0 Karma

harsmarvania57
Ultra Champion

I have tested below configuration in inputs.conf in my lab environment

inputs.conf

[http://test-token]
disabled = 0
token = XXX-XXXXX-XXX-XXXX
persistentQueueSize = 100MB

and after that I can see that pq_<token> file generated in $SPLUNK_HOME/var/run/splunk/httpin/ which means that it supports persistentQueue

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

ICYMI - Check out the latest releases of Splunk Edge Processor

Splunk is pleased to announce the latest enhancements to Splunk Edge Processor.  HEC Receiver authorization ...

Introducing the 2024 SplunkTrust!

Hello, Splunk Community! We are beyond thrilled to announce our newest group of SplunkTrust members!  The ...