Getting Data In

After forwarder network goes down and is restored, why does only one indexer receive lost data?

New Member


I have data cloning to 2 splunk indexers (instances):

                  /          \
             Splunk01     Splunk02

when the network goes out on the forwarder1, splunk01 and splunk02 don't receive data. Which is expected. The problem is when, once the network is restored, splunk01 gets the lost data, but splunk02 does not get the data that was lost.

my forwarder outputs.conf is (server names have been changed to make this easier to understand):
defaultGroup = firstsplunkserver,secondsplunkserver

server = splunk01:9997

server = splunk02:9997

Why isn't splunk02 getting the lost data? How do you clone this data from splunk01?


0 Karma


I can't find anything that goes along with this issue. Have you run wireshark/tcpdump on splunk02 to view packets coming in, or on your forwarder to view packets going out, and verify that nothing is getting destined to splunk02? Or, are you just searching the data and not seeing it? Reason I ask is to determine which side of the connection is becoming a problem.

I would also search the forwarder and splunk02's splunkd.log to see if anything comes up during that time period indicating one side or the other.

I'll keep looking, but the way you're listing these servers in the defaultGroup should always clone the data to anything there.

0 Karma


Not that I know of. Did you try reversing the order so 02 is first in the list? See if the data goes to it and not 01, or if 01 is the only one capable of receiving this data? That would be the last test I would try to pin point the actual issue.

0 Karma

New Member

I'm searching the data and not seeing it. On the splunk forwarder it just says it disconnects and reconnects to that server.

is there anything i need to enable on the splunk02 instance?

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...