Getting Data In

After forwarder network goes down and is restored, why does only one indexer receive lost data?

New Member


I have data cloning to 2 splunk indexers (instances):

                  /          \
             Splunk01     Splunk02

when the network goes out on the forwarder1, splunk01 and splunk02 don't receive data. Which is expected. The problem is when, once the network is restored, splunk01 gets the lost data, but splunk02 does not get the data that was lost.

my forwarder outputs.conf is (server names have been changed to make this easier to understand):
defaultGroup = firstsplunkserver,secondsplunkserver

server = splunk01:9997

server = splunk02:9997

Why isn't splunk02 getting the lost data? How do you clone this data from splunk01?


0 Karma


I can't find anything that goes along with this issue. Have you run wireshark/tcpdump on splunk02 to view packets coming in, or on your forwarder to view packets going out, and verify that nothing is getting destined to splunk02? Or, are you just searching the data and not seeing it? Reason I ask is to determine which side of the connection is becoming a problem.

I would also search the forwarder and splunk02's splunkd.log to see if anything comes up during that time period indicating one side or the other.

I'll keep looking, but the way you're listing these servers in the defaultGroup should always clone the data to anything there.

0 Karma


Not that I know of. Did you try reversing the order so 02 is first in the list? See if the data goes to it and not 01, or if 01 is the only one capable of receiving this data? That would be the last test I would try to pin point the actual issue.

0 Karma

New Member

I'm searching the data and not seeing it. On the splunk forwarder it just says it disconnects and reconnects to that server.

is there anything i need to enable on the splunk02 instance?

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...