All Apps and Add-ons

When HF with "Splunk DB Connect" send data to Indexer, and if Indexer is down, the data will be lost?

yutaka1005
Builder

If there is a environment like below, if Indexer is down, the data will be lost?

*HF with Splunk DB Connect
*Indexer

I think data input like "file monitor","data from other Universal Forwarder" etc will stop, if indexer is down, so the data won't be lost.

But I have to configure persistent queue to avoid data lost, if I use data input such "TCP","scripted input".

which type is data input of Splunk DB Connet?
Do I have to configure persistent queue?
Or Splunk DB Connet automatically stop, if outputqueue is filled?

If someone know about it , please tell me.

0 Karma

harsmarvania57
Ultra Champion

Splunk DB Connect 3 is using HEC (HTTP Event Collector) to ingest data and based on how indexing works diagram https://wiki.splunk.com/Community:HowIndexingWorks, HEC uses httpinputq but I am not able to find anything related to httpinpuq in Splunk Docs and not sure how DB connect on HFW will react when Indexer is down.

EDIT: However based on https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Inputsconf#HTTP_Event_Collector_.28HEC.29_-... , you can set persistentQueueSize for [http://<token>]

0 Karma

harsmarvania57
Ultra Champion

I have tested below configuration in inputs.conf in my lab environment

inputs.conf

[http://test-token]
disabled = 0
token = XXX-XXXXX-XXX-XXXX
persistentQueueSize = 100MB

and after that I can see that pq_<token> file generated in $SPLUNK_HOME/var/run/splunk/httpin/ which means that it supports persistentQueue

0 Karma
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...