We have a batch script which monitors files under some folder and then creates a log file with the file name and file creation time information every 5 mins. Currently, that batch script is creating an empty log file as there are no files in the inspected folder, but there are events coming from that log file to Splunk every 5 minutes.
Is it possible that the Splunk forwarder caches old events and re-sends again and again?
It started at noon and Splunk was getting until 12:55 PM however it stopped after 1 PM. Does this mean anything?
That sounds unlikely as long as indexers are working fine. What's the source field of the repeated data? Can you post the input configuration for that source?
If your indexers are not working fine and you use
useAck then there is a small chance of duplicates: http://docs.splunk.com/Documentation/Splunk/6.2.2/Forwarding/Protectagainstlossofin-flightdata#The_p...
...not regularly every five minutes though, and this would be logged in the forwarder's splunkd.log.
After sending a data block, the forwarder maintains a copy of the data in its wait queue until it receives an acknowledgment. In the meantime, it continues to send additional blocks as usual. If the forwarder doesn't get acknowledgment for a block within 300 seconds (by default), it closes the connection. You can change the wait time by setting the readTimeout attribute in outputs.conf.