Splunk Search

How to extract the file type from my logs and get the count of successfully processed files by file type?



I am searching for some way to extract count of each file type which is successfully processed. The logs contain various levels of processing of file, but the final line contains status of the processing. I need to extract all such successful processed files and count how many successful files there are for each type. For example, below is my pattern in the log.

11:44:01,982 some other log here File: ../../../transfer/dir/path/**MYFILETYPE**.SOMEOTHERPATTEN.xml successfully imported.

So I need to extract all such lines and build a table with count of each file type. Can someone please help with what regex I need to use to get this?

0 Karma

Revered Legend

Give this a try

index=foo sourcetype=bar "successfully imported" | rex "File:\s+.*\/(?<filetype>\w+)\..*\ssuccessfully imported" | stats count by filetype
0 Karma

Splunk Employee
Splunk Employee

Hey boddunan,

First of all, I'd search for all data where "successfully imported." exists:

your base search... | search "successfully imported."

Then I'd add a regular expression like this:

| rex field=_raw "path\/(?<file_name>[A-Za-z0-9*%$]*\.\w+\.\w{2,4})"

You should then see a field in your data called "file_name". You should then be able to do this:

your base search... | search "successfully imported."| rex field=_raw "path\/(?<file_name>[A-Za-z0-9*%$]*\.\w+\.\w{2,4})" | stats count by file_name

Let me know how that goes, If you want to pull apart the regular expression and see how it works(or how you can debug it). Try www.regex101.com.

Copy the path\/(?<file_name>[A-Za-z0-9*%$]*\.\w+\.\w{2,4}) in the top input box and your data underneath. On the right hand side it'll show you how it works.



This should work

index=* | rex "File:\s(?<filename>[^\s])\s(?<status>successfully)" | stats count by filename
0 Karma
.conf21 Now Fully Virtual!
Register for FREE Today!

We've made .conf21 totally virtual and totally FREE! Our completely online experience will run from 10/19 through 10/20 with some additional events, too!