topic Re: Pull out fields embedded in logs in Getting Data In

Pull out fields embedded in logs

mdavis43 — Wed, 09 Oct 2013 18:19:24 GMT

I'm looking for information about how to pull out field information from inside the log messages. For example...

Message=(Error) I/O error on file system 'prodops' operation WRITE inode (Message repeated 4732 times)

Message=(Error) I/O error on file system 'proxy' operation WRITE inode (Message repeated 4 times)

Message=(Error) I/O error on file system 'wwtowip' operation WRITE inode

These come from Windows event logs and I want to be able to sort on how many times these errors happen to each individual filesystem.

Re: Pull out fields embedded in logs

lukejadamec — Wed, 09 Oct 2013 18:31:46 GMT

You should use rex to create a field from within the Message field that you can search on later.

index=main sourcetype="*security*" | rex field=Message "... error on file system '(?<filesystem>.*)' | stats count by filesystem

Re: Pull out fields embedded in logs

lguinn2 — Wed, 09 Oct 2013 18:41:04 GMT

There are a number of ways to extract fields. Overview of search-time field extractions is a good resource.

For the data you have here, you could do this in the rex command as well -

yoursearchhere
| rex "(?<msg>.*?file system)\s*\'(?<file_system>.*?)\' operation (?<operation>.*)(?:\(Message repeated (?<msgcount>\d+)  times\))*"
| fillnull value=1 msgcount
| stats sum(msgcount) as MessageCount by msg file_system

I've probably made some typo in the regular expression... but I hope you get the idea

Re: Pull out fields embedded in logs

mdavis43 — Wed, 09 Oct 2013 19:24:27 GMT

Thanks! This worked great inline.

Re: Pull out fields embedded in logs

mdavis43 — Wed, 09 Oct 2013 19:24:45 GMT

Thanks, this looks great for long term usage.