Yes, there are a couple of ways. The core problem here is very simple, so maybe mentioning it again will bring to mind other ways to fix this. The problem is, that zip files get read and ingested *in their entirety* every time they get read and ingested. That's why you have duplicated data. Because the 30 rows in that zip file now, when the device saves a new copy with another 5 rows in it, those 35 rows ALL get read again and ingested. Because it's a zip file, and that's how it works. So, you need a solution that makes this zip-file-append-behavior not be an issue. What's right for your environment, or easiest, isn't anything for me to tell you. But here are options. 1) You could unzip the file via some other method (batch file/script), into another folder and have Splunk monitor that other unzipped file in that other folder. Then a standard monitor input will work on that new location/file, because Splunk can generally handle this. Though you might have to play with crcsalt and other similar settings a bit. 2) Or maybe you could try the batch/sinkhole input again, only this time use an absolute path. Then the batch/sinkhole input will delete it each time and there will be no re-reading of the old contents of the file because they won't be there (as far as I know the path requirements between monitor:: and batch:: inputs are identical and you literally shouldn't need to change anything except "monitor" to "batch" and add the move_policy=sinkhole to it, but if it's complaining about absolute paths, well, change it to absolute pathing then.) 3) Reconfigure the sending/saving device to not zip the file it's saving and instead leave it plaintext. If you can do this, then a standard monitor input will probably work on the non-zipped file because monitor inputs are really smart about appended data on "regular" files. 4) Reconfigure the sending/saving device to not append to an existing file when it builds a zip file. Then a standard monitor input will work, because it won't be an appended-to file, but instead "new from empty". I'm sure there are probably other solutions too. If you continue to have problems out of the batch/sinkhole input, please post your input stanza here so we can take a look at it - it's probably some other typo not letting you do this. In any case, do let us know what ended up working! -Rich
... View more
