All Apps and Add-ons

When HF with "Splunk DB Connect" send data to Indexer, and if Indexer is down, the data will be lost?

yutaka1005
Builder

If there is a environment like below, if Indexer is down, the data will be lost?

*HF with Splunk DB Connect
*Indexer

I think data input like "file monitor","data from other Universal Forwarder" etc will stop, if indexer is down, so the data won't be lost.

But I have to configure persistent queue to avoid data lost, if I use data input such "TCP","scripted input".

which type is data input of Splunk DB Connet?
Do I have to configure persistent queue?
Or Splunk DB Connet automatically stop, if outputqueue is filled?

If someone know about it , please tell me.

0 Karma

harsmarvania57
Ultra Champion

Splunk DB Connect 3 is using HEC (HTTP Event Collector) to ingest data and based on how indexing works diagram https://wiki.splunk.com/Community:HowIndexingWorks, HEC uses httpinputq but I am not able to find anything related to httpinpuq in Splunk Docs and not sure how DB connect on HFW will react when Indexer is down.

EDIT: However based on https://docs.splunk.com/Documentation/Splunk/7.2.3/Admin/Inputsconf#HTTP_Event_Collector_.28HEC.29_-... , you can set persistentQueueSize for [http://<token>]

0 Karma

harsmarvania57
Ultra Champion

I have tested below configuration in inputs.conf in my lab environment

inputs.conf

[http://test-token]
disabled = 0
token = XXX-XXXXX-XXX-XXXX
persistentQueueSize = 100MB

and after that I can see that pq_<token> file generated in $SPLUNK_HOME/var/run/splunk/httpin/ which means that it supports persistentQueue

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Calling All Security Pros: Ready to Race Through Boston?

Hey Splunkers, .conf25 is heading to Boston and we’re kicking things off with something bold, competitive, and ...

Beyond Detection: How Splunk and Cisco Integrated Security Platforms Transform ...

Financial services organizations face an impossible equation: maintain 99.9% uptime for mission-critical ...

Customer success is front and center at .conf25

Hi Splunkers, If you are not able to be at .conf25 in person, you can still learn about all the latest news ...