Getting Data In

How to view status of indexing of evtx files in a folder

rockb
Explorer

I am using Splunk to review logs from disconnected systems.  We have the users export the evtx files and send them to us.  I then put them in a folder and Splunk indexes the new files.  

Is there an easy way to see the indexing process?  Right now I just keep hitting refresh occasionally until nothing changes.

Labels (1)
0 Karma

Azeemering
Builder

The easiest way to check if a file has finished being ingested is to check the _internal index like this:

 

index=_internal sourcetype=splunkd ArchiveProcessor yourfilename.evtx

 

The order of internal logging is

1. Handling FIle
2. Reading Path
3. Processing xxxx
4. Finished processing file

It will contains entries like this. You will see the message Finished processing file when the file has been ingested completely. (I used a file called sectest.evtx)

10-05-2021 20:53:52.975 +0200 INFO ArchiveProcessor [17620 archivereader] - Finished processing file 'H:\evtx\sectest.evtx', removing from stats
10-05-2021 20:53:52.975 +0200 INFO ArchiveProcessor [17620 archivereader] - Processing windows event log file H:\evtx\sectest.evtx
10-05-2021 20:53:52.975 +0200 INFO ArchiveProcessor [17620 archivereader] - reading path=H:\evtx\sectest.evtx (seek=0 len=21041152)
10-05-2021 20:53:52.973 +0200 INFO ArchiveProcessor [17620 archivereader] - Handling file=H:\evtx\sectest.evtx

Obviously you can set an alert, or some type of message when you see that message come by.

 

0 Karma

rockb
Explorer

Thank you for the reply. 

 

There are 18 evtx files in a single folder.  Is there a way to get status of indexing of the entire folder?

0 Karma

Azeemering
Builder

It will follow the same principal as the previous example.

For each individual file it will go through that process. So you could create a saved search with a stats count command that count the number of "Finished processing file" messages.
Try and be creative and use Splunk _internal event logging to your own advantage.

Another brainfart might be to send an alert when you are not seeing "Finished processing file" anymore in the _internal logging. This might also be an indicator that all files have been processed.

0 Karma
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.

Can’t make it to .conf25? Join us online!

Get Updates on the Splunk Community!

Leveraging Automated Threat Analysis Across the Splunk Ecosystem

Are you leveraging automation to its fullest potential in your threat detection strategy?Our upcoming Security ...

Can’t Make It to Boston? Stream .conf25 and Learn with Haya Husain

Boston may be buzzing this September with Splunk University and .conf25, but you don’t have to pack a bag to ...

Splunk Lantern’s Guide to The Most Popular .conf25 Sessions

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...