I am using Splunk to review logs from disconnected systems. We have the users export the evtx files and send them to us. I then put them in a folder and Splunk indexes the new files.
Is there an easy way to see the indexing process? Right now I just keep hitting refresh occasionally until nothing changes.
The easiest way to check if a file has finished being ingested is to check the _internal index like this:
index=_internal sourcetype=splunkd ArchiveProcessor yourfilename.evtx
The order of internal logging is
1. Handling FIle
2. Reading Path
3. Processing xxxx
4. Finished processing file
It will contains entries like this. You will see the message Finished processing file when the file has been ingested completely. (I used a file called sectest.evtx)
10-05-2021 20:53:52.975 +0200 INFO ArchiveProcessor [17620 archivereader] - Finished processing file 'H:\evtx\sectest.evtx', removing from stats
10-05-2021 20:53:52.975 +0200 INFO ArchiveProcessor [17620 archivereader] - Processing windows event log file H:\evtx\sectest.evtx
10-05-2021 20:53:52.975 +0200 INFO ArchiveProcessor [17620 archivereader] - reading path=H:\evtx\sectest.evtx (seek=0 len=21041152)
10-05-2021 20:53:52.973 +0200 INFO ArchiveProcessor [17620 archivereader] - Handling file=H:\evtx\sectest.evtx
Obviously you can set an alert, or some type of message when you see that message come by.
Thank you for the reply.
There are 18 evtx files in a single folder. Is there a way to get status of indexing of the entire folder?
It will follow the same principal as the previous example.
For each individual file it will go through that process. So you could create a saved search with a stats count command that count the number of "Finished processing file" messages.
Try and be creative and use Splunk _internal event logging to your own advantage.
Another brainfart might be to send an alert when you are not seeing "Finished processing file" anymore in the _internal logging. This might also be an indicator that all files have been processed.