Hi @kthudi6,
Did you resolve this? You mentioned duplicate data/log files. This looks like you have a rotating log file and you're indexing both the actual log file and the rotated logs together?
If this is the case, you'll probably want to setup your file monitoring logic to monitor specific files and/or update your log rolling logic (place files in a different location, have an archive script to clean out old files, etc.).
If you monitor a file that's being constantly written to, Splunk may keep re-ingesting the data from that file because the CRC and End of File keep changing before Splunk gets to the end of file. In this case, if a delay in data is okay, you can only monitor the rolled log files, or create logic to copy the data from the active log file to a separate file that is monitored by Splunk (i.e. copy every 30 seconds or 1 minute).
... View more