How to extract the file type from my logs and get the count of successfully processed files by file type?

[boddunan]

Hi,

I am searching for some way to extract count of each file type which is successfully processed. The logs contain various levels of processing of file, but the final line contains status of the processing. I need to extract all such successful processed files and count how many successful files there are for each type. For example, below is my pattern in the log.

11:44:01,982 some other log here File: ../../../transfer/dir/path/**MYFILETYPE**.SOMEOTHERPATTEN.xml successfully imported.

So I need to extract all such lines and build a table with count of each file type. Can someone please help with what regex I need to use to get this?

[somesoni2]

Give this a try

index=foo sourcetype=bar "successfully imported" | rex "File:\s+.*\/(?<filetype>\w+)\..*\ssuccessfully imported" | stats count by filetype

[ktugwell_splunk]

Hey boddunan,

First of all, I'd search for all data where "successfully imported." exists:

your base search... | search "successfully imported."

Then I'd add a regular expression like this:

| rex field=_raw "path\/(?<file_name>[A-Za-z0-9*%$]*\.\w+\.\w{2,4})"

You should then see a field in your data called "file_name". You should then be able to do this:

your base search... | search "successfully imported."| rex field=_raw "path\/(?<file_name>[A-Za-z0-9*%$]*\.\w+\.\w{2,4})" | stats count by file_name

Let me know how that goes, If you want to pull apart the regular expression and see how it works(or how you can debug it). Try www.regex101.com.

Copy the path\/(?<file_name>[A-Za-z0-9*%$]*\.\w+\.\w{2,4}) in the top input box and your data underneath. On the right hand side it'll show you how it works.

Thanks

[sundareshr]

This should work

index=* | rex "File:\s(?<filename>[^\s])\s(?<status>successfully)" | stats count by filename