So after talking about this with tech support and looking at my own system a bunch I have figured out what is going on. Splunk is designed such that is will not re-index rolled over logs. Even though the file name changes it realizes that the log file has already been indexed. This document talks about how Splunk handles log file rotation and doesn't re-index the data. That is why it is ok for them to say [monitor://$SPLUNK_HOME/var/log/splunk] instead of [monitor://$SPLUNK_HOME/var/log/splunk/*.log] In fact the first one is right and here is why. Log file rotation. What I mean by that is that when the metrics.log file rotates Splunk has not had a chance to get the tail of that log indexed. So, when metrics.log is renamed to metrics.log.1 Splunk looks at the file and realizes that is has already indexed much of that file, but there is a bit at the end that has not been index. So, it indexes that part and since the file name is not metrics.log.1 and not metrics.log that is what is reflected in the source. If the monitor stanza was set to only pull in *.log then the tail portion that was not fully indexed would never get pulled in as metrics.log.1 would not match the file mask *.log. I verified this by looking at some of the data that was indexed in metrics.log.1 and found no corresponding entries in a metrics.log source. So, it isn't duplicating the data and the world is still turning just fine. ... View more