Using indexer discovery, how to check if a forwarder has forwarded a file to the indexer cluster?

[guotao4321]

Issue:
- After uploading file to forwarder monitoring directory, we cannot search it on search head.
Environment:
- 1 search head --> 1 indexer cluster {1 master + 3 indexers} <-- 1 universal forwarder
- enable "Forward master node data to the indexer layer": http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/Forwardmasterdata
- configure "Use indexer discovery to connect forwarders to peer nodes": http://docs.splunk.com/Documentation/Splunk/6.5.0/Indexer/indexerdiscovery

splunkd.log on Forwarder:

11-24-2016 11:07:24.347 +0800 INFO TcpOutputProc - Closing stream for idx=172.16.1.81:9997
11-24-2016 11:07:24.348 +0800 INFO TcpOutputProc - Connected to idx=172.16.1.82:9997 using ACK.
11-24-2016 11:07:38.544 +0800 INFO TailReader - Archive file='/data/tutorialdata.zip' updated less than 10000ms ago, will not read it until it stops changing. File size=0
11-24-2016 11:07:48.598 +0800 INFO TailReader - Archive file='/data/tutorialdata.zip' has stopped changing, will read it now.
11-24-2016 11:07:48.598 +0800 INFO ArchiveProcessor - Handling file=/data/tutorialdata.zip
11-24-2016 11:07:48.598 +0800 INFO ArchiveProcessor - new tailer already processed path=/data/tutorialdata.zip
11-24-2016 11:07:54.207 +0800 INFO TcpOutputProc - Closing stream for idx=172.16.1.82:9997

11-24-2016 11:07:54.207 +0800 INFO TcpOutputProc - Connected to idx=172.16.1.81:9997 using ACK.

Findings:
1. the forwarder has already handled the file. How can we check if it successfully forwards it to the indexer cluster?
2. the forwarder is continuing to change the connected indexers. Is it normal or an issue of the communication between the forwarder and indexers?

Thank you very much for helps.

[lguinn2]

The forwarder will continue to change the connected indexer. That is called "auto load balancing" and it is the desired behavior. It is also the default.

If you want to know if the file has arrived on the indexer, you only need to search for it:

index=* source="/data/tutorialdata.zip"

If the file does not appear when you search, check to see what index was used in the inputs.conf on the forwarder. Make sure that index exists on the indexers and that you have permission to read it.

[guotao4321]

Thanks for the reply. Glad to know that changing connected indexer is a normal behavior, so it's easy to troubleshoot this issue. We tried other file price.csv.zip and run the search * source="/data/price.csv.zip". IT WORKS. Therefore we think it is the issue about the file.

Actually, when we create the indexers in the cluster, we clone a previous distrubuted index where we had forwarded the tutorialdata.zip. Although we remove all the database on the new indexer, will it save the hash or something else to mark the file forwarded? When we forward tutuorialdata.zip again, the indexer will ignore it by checking the hash？

If it is, how can we clean the hash records to make the new indexer working for a duplicate file?

Thank you very much.

Regards,
Tao