Pull out fields embedded in logs

[mdavis43]

I'm looking for information about how to pull out field information from inside the log messages. For example...

Message=(Error) I/O error on file system 'prodops' operation WRITE inode (Message repeated 4732 times)

Message=(Error) I/O error on file system 'proxy' operation WRITE inode (Message repeated 4 times)

Message=(Error) I/O error on file system 'wwtowip' operation WRITE inode

These come from Windows event logs and I want to be able to sort on how many times these errors happen to each individual filesystem.

[lguinn2 · Accepted solution]

There are a number of ways to extract fields. Overview of search-time field extractions is a good resource.

For the data you have here, you could do this in the rex command as well -

yoursearchhere
| rex "(?<msg>.*?file system)\s*\'(?<file_system>.*?)\' operation (?<operation>.*)(?:\(Message repeated (?<msgcount>\d+)  times\))*"
| fillnull value=1 msgcount
| stats sum(msgcount) as MessageCount by msg file_system 

I've probably made some typo in the regular expression... but I hope you get the idea

[mdavis43]

Thanks, this looks great for long term usage.

[lukejadamec]

You should use rex to create a field from within the Message field that you can search on later.

index=main sourcetype="*security*" | rex field=Message "... error on file system '(?<filesystem>.*)' | stats count by filesystem

[mdavis43]

Thanks! This worked great inline.