I'm trying to use "Monitor Files & Directories" as data input. I got two Data Input sources,
The incoming files are of "csv" type and have unique file name (timestamp in the file name). I see only the first csv file getting indexed and not the subsequent ones that are generated by the script. I've read http://answers.splunk.com/questions/4103/directory-monitoring-not-picking-up-new-files and http://www.splunk.com/base/Documentation/latest/Admin/Monitorfilesanddirectories, but not sure what else I need to do. Few Questions,
In the documentation it says the monitor would only check for new files every 24 hours - is that right? How else can I make it to continously look for new files in the directly? Do I need to use crawl?
Is it possible to use monitor to do the above and when the file is indexed delete that file (similar to using sinkhole)?
In my case once a file is copied into the directory it's not changed, so I basically just want to delete it once Splunk has indexed it.
/opt/splunk/var/spool/splunkwhich acts as a sinkhole. It will index everything you want then delete the files
However, i think you should look into why your monitor is not reading the additional csv files that are being created. Check your `splunkd.log` for any logs related to this. Perhaps the files are too similar and you are getting a crccheck issue (where the crc of the files is too similar and splunk doesnt index because it thinks its the same file. Basically the first 250 chars of the files are the same, in this case look for `crcSalt` in `inputs.conf`)
Please read input.conf.spec for more information on
spool directory is like big brother, always watching for files being dropped there. Once it reads the file, it deletes it. Think of it as a sinkhole.. You can always try it out, dump some files there and youll experience it first hand. The only issue with this method, i think, is that you cant really specify source, sourcetype, which index the data should go to etc.. I believe 4.2 will have some improvements in this area.
Thanks. I added "crcSalt =
My other issue is how do I delete the indexed files and still keep continous inputs. The "batch" option seems to work only one-time. Would writing to spool directory continuously read new files - just like monitor?