topic Re: How to detect that a file has been indexed in Getting Data In

How to detect that a file has been indexed

JeremyHagan — Wed, 06 Nov 2013 21:56:18 GMT

I have an automated process running on a Windows server that has the Universal Forwarder installed. It drops files for indexing in a specific folder and I want it to be able to clean up and delete files that have been indexed.

In my testing I found that the splunkd.log file had entries like this:

11-06-2013 22:08:49.817 +1100 INFO
BatchReader - Removed from queue
file='U:\WebsenseExport\Exported\20131106100010.csv'.

My script was parsing this file and if it found this entry it would delete the file. This morning I find a bunch of files have been indexed, but are not noted in the splunkd.log. I suspect that this is because they are small and the "BatchReader" doesn't handle them? This is just a guess.

Is there a foolproof way I can tell, from the forwarder, that a file has definitely been indexed?

Re: How to detect that a file has been indexed

somesoni2 — Wed, 06 Nov 2013 22:05:42 GMT

you might check your Index where the data is indexed, with the source=yourfilename and see if any events are found there.

Re: How to detect that a file has been indexed

JeremyHagan — Wed, 06 Nov 2013 22:11:30 GMT

Thanks for the answer, but I need to be able to check from the forwarder and programmatically.

Your suggestion would require access to the splunk indexer.

Re: How to detect that a file has been indexed

MuS — Thu, 07 Nov 2013 10:18:26 GMT

Hi JeremyHagan,

you can use the REST end point

 /services/admin/inputstatus/TailingProcessor:FileStatus

to track if a universal forwarder is reading files monitored or completed sending events.

In the end point you can find information about "open file", and others showing "finished reading".

Some details about the endpoint information, when the percent is 100% :

"finished reading" means that the file has been read and forwarded till the end.

"open file" means the same, but in addition the handle on the file is still open (because it has been less than 3 seconds, or because it is being 'tailed', or the file has just being reopen for any update or rotation).

Splunk will monitor every file, because Splunk assumes that a new event can be added to any file.

hope this helps...

cheers, MuS

Re: How to detect that a file has been indexed

lguinn2 — Fri, 08 Nov 2013 00:09:33 GMT

I like the answer from @MuS

To follow up, you might also take a look at this blog post. It's a bit old, but you can grab some code that does something like what you want...

I love code samples

http://blogs.splunk.com/2011/01/02/did-i-miss-christmas-2/