How to detect that a file has been indexed

[JeremyHagan]

I have an automated process running on a Windows server that has the Universal Forwarder installed. It drops files for indexing in a specific folder and I want it to be able to clean up and delete files that have been indexed.

In my testing I found that the splunkd.log file had entries like this:

11-06-2013 22:08:49.817 +1100 INFO
BatchReader - Removed from queue
file='U:\WebsenseExport\Exported\20131106100010.csv'.

My script was parsing this file and if it found this entry it would delete the file. This morning I find a bunch of files have been indexed, but are not noted in the splunkd.log. I suspect that this is because they are small and the "BatchReader" doesn't handle them? This is just a guess.

Is there a foolproof way I can tell, from the forwarder, that a file has definitely been indexed?

[MuS]

Hi JeremyHagan,

you can use the REST end point

 /services/admin/inputstatus/TailingProcessor:FileStatus

to track if a universal forwarder is reading files monitored or completed sending events.

In the end point you can find information about "open file", and others showing "finished reading".

Some details about the endpoint information, when the percent is 100% :

"finished reading" means that the file has been read and forwarded till the end.

"open file" means the same, but in addition the handle on the file is still open (because it has been less than 3 seconds, or because it is being 'tailed', or the file has just being reopen for any update or rotation).

Splunk will monitor every file, because Splunk assumes that a new event can be added to any file.

hope this helps...

cheers, MuS

[lguinn2]

I like the answer from @MuS

To follow up, you might also take a look at this blog post. It's a bit old, but you can grab some code that does something like what you want...

I love code samples

http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/

[JeremyHagan]

Thanks for the answer, but I need to be able to check from the forwarder and programmatically.

Your suggestion would require access to the splunk indexer.

[somesoni2]

you might check your Index where the data is indexed, with the source=yourfilename and see if any events are found there.