Why are we receiving a massive amount of duplicate events from our forwarder, resulting in 4 license violations?

[ekst_andwii]

We are experiencing a massive duplication of events in two log files indexed by Splunk. This started suddenly on a Friday and went on all weekend, resulting in 4 license violations. There are other log files indexed by the same forwarder and they share monitor and other config.

Example of line in the log file:

|2016-09-12 10:59:59,597|DEBUG|HikariCP connection filler (pool HikariPool-0)|HikariPool|After fill pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)

In Splunk (event source):

...
|2016-09-12 10:59:59,597|DEBUG|HikariCP connection filler (pool HikariPool-0)|HikariPool|After fill pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
|2016-09-12 10:59:59,563|DEBUG|Hikari Housekeeping Timer (pool HikariPool-0)|HikariPool|Before cleanup pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
|2016-09-12 10:59:59,563|DEBUG|Hikari Housekeeping Timer (pool HikariPool-0)|HikariPool|After cleanup pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
|2016-09-12 10:59:59,597|DEBUG|HikariCP connection filler (pool HikariPool-0)|HikariPool|After fill pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
|2016-09-12 10:59:59,563|DEBUG|Hikari Housekeeping Timer (pool HikariPool-0)|HikariPool|Before cleanup pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
|2016-09-12 10:59:59,563|DEBUG|Hikari Housekeeping Timer (pool HikariPool-0)|HikariPool|After cleanup pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
|2016-09-12 10:59:59,597|DEBUG|HikariCP connection filler (pool HikariPool-0)|HikariPool|After fill pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
|2016-09-12 10:59:59,563|DEBUG|Hikari Housekeeping Timer (pool HikariPool-0)|HikariPool|Before cleanup pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
|2016-09-12 10:59:59,563|DEBUG|Hikari Housekeeping Timer (pool HikariPool-0)|HikariPool|After cleanup pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
|2016-09-12 10:59:59,597|DEBUG|HikariCP connection filler (pool HikariPool-0)|HikariPool|After fill pool stats HikariPool-0 (total=20, inUse=3, avail=17, waiting=0)
...

619 events in total for timestamp 2016-09-12 10:59:59,597

Any ideas??

[ddrillic]

Are you sure it's the fault of Splunk? - ; )

Similar case at How do I fix a large amount of duplicate events that are locking out my instance?

It says -

-- Apologies for the confusion. This answer is all set. Further investigation showed that our product's logs were guilty of producing duplicate entries. Thank you for your help!

BTW, the best practice is to avoid DEBUG logging as they inflate the logs, or, if truly needed, to minimize the time the DEBUG logging is being used.

[ekst_andwii]

Yes, we double checked. There is only one line of log for the given timestamp.

[lukejadamec]

I've seen this before on a Linux system when the monitored directory was set up as a mirror so it was constantly being re-written and re-indexed.

[ekst_andwii]

I don't think there is any mirroring on the directory. Plain Linux installation (on a VM) with Tomcat.

[lukejadamec]

Can you post your inputs.conf settings for this source?
Can you also verify that they are all from the same source? The events you posted do not show source or sourcetype.

[ekst_andwii]

I appreciate your reply but I have submitted a support case to Splunk regarding this. I will let you know what the outcome is.

99% of the events are from the same source, a few(!) are from .1 files (rolled over logs).

[jwelch_splunk]

Are you using Indexer Clustering and IndexerDiscovery?

[ekst_andwii]

No, this is pretty much a default installation. We use whitelists to include logs in the monitors.