Sourcetype - checkpoint -aad :aure

[arunsundarm]

I Am having Hf and it is configured to send data via sourcetype A

After sometime it stops sending data to A

Then i move the data to diffrent HF in sourcetype : test ( to test if it is working) 

then from new HF I am routing the data to Source type A itself

Will it reingest the data or checkpoint from the data it is left off, will it ignore the data which was sent to sourcetype :test?? need help and clear explanation

[PickleRick]

Each ingested event is a separate entity and is processed independently so if you make the same data available to the input twice (for example by sending the same syslog event to a network-listening input) it's gonna get ingested, processed and indexed twice.

It's up to the input - if applicable - to make sure the same data is not ingested twice. That's why file monitoring inputs have some logic implemented which keeps track which files and "how far" have been read so far, database inputs can have checkpoints storing information at which point in time you stopped reading from DB and so on. But that happens on the input level.

After the even is read by the input, it's getting processed regardless of whether another "copy" of it have ever been indexed or not.

[richgalloway]

The two HFs have no way to know what the other has done so the new HF probably will reingest the same data.  I say "probably" because I'm not familiar with the mechanism the add-on uses to fetch data from Azure.  If the checkpoint is stored on the HF then data will be reingested by a different HF; if the checkpoint is stored on Azure then data may not be reingested.

[jconger]

I can confirm that the checkpoint data is stored in the KV Store on the forwarder.  The checkpoint is the last timestamp retrieved from the Azure REST API.  So if you use a new forwarder, the data will be ingested again (duplicate data).