After forwarder network goes down and is restored, why does only one indexer receive lost data?

[echonest_krystl]

Hi,

I have data cloning to 2 splunk indexers (instances):

                   forwarder1
                  /          \
             Splunk01     Splunk02

when the network goes out on the forwarder1, splunk01 and splunk02 don't receive data. Which is expected. The problem is when, once the network is restored, splunk01 gets the lost data, but splunk02 does not get the data that was lost.

my forwarder outputs.conf is (server names have been changed to make this easier to understand):
[tcpout]
defaultGroup = firstsplunkserver,secondsplunkserver

[tcpout:firstsplunkserver]
server = splunk01:9997

[tcpout:secondsplunkserver]
server = splunk02:9997

Why isn't splunk02 getting the lost data? How do you clone this data from splunk01?

Thanks!

[hortonew]

I can't find anything that goes along with this issue. Have you run wireshark/tcpdump on splunk02 to view packets coming in, or on your forwarder to view packets going out, and verify that nothing is getting destined to splunk02? Or, are you just searching the data and not seeing it? Reason I ask is to determine which side of the connection is becoming a problem.

I would also search the forwarder and splunk02's splunkd.log to see if anything comes up during that time period indicating one side or the other.

I'll keep looking, but the way you're listing these servers in the defaultGroup should always clone the data to anything there.

[hortonew]

Not that I know of. Did you try reversing the order so 02 is first in the list? See if the data goes to it and not 01, or if 01 is the only one capable of receiving this data? That would be the last test I would try to pin point the actual issue.

[echonest_krystl]

I'm searching the data and not seeing it. On the splunk forwarder it just says it disconnects and reconnects to that server.

is there anything i need to enable on the splunk02 instance?