I started noticing some duplicate events in my logs recently. As I was curious to know what was happening, I searched in the dbx logs to see if something went wrong around the time I could see the duplicates. I saw the message "WARN:Shutdown - Performing Java Bridge Server shutdown... " in the logs close to the time when the duplicates occurred and I guess this message was logged due to splunk restart.
After taking a closer look at the logs, my understanding is that whenever splunk is restarted, the last rising column value recorded before the Java Bridge Server shutdown is somehow lost, and when splunk is up and running again, it doesn't see the actual latest rising column value. Instead, it sees the value before that. Hence, if there are new events between the two rising column time period, they will get indexed again. The duplicates that I see in the logs supports this assumption.
Example is:
3/25/15 3:31:00.019 PM Applying latest tail.rising.column value=2015-03-25 15:27:49.0
3/25/15 3:29:00.019 PM Applying latest tail.rising.column value=2015-03-25 15:25:22.0
3/25/15 3:25:00.019 PM WARN:Shutdown - Performing Java Bridge Server shutdown...
3/25/15 3:24:00.018 PM Applying latest tail.rising.column value=2015-03-25 15:21:53.0
3/25/15 3:22:00.020 PM Applying latest tail.rising.column value=2015-03-25 15:19:55.0
In the above scenario, when splunk restarts, all the events with last_modified_time > 2015-03-25 15:19:55.0 will be indexed which causes duplicates if there are new events between 2015-03-25 15:21:53.0 and 2015-03-25 15:19:55.0.
Anyone came across such an issue ??
