Getting Data In

Pull out fields embedded in logs

mdavis43
Path Finder

I'm looking for information about how to pull out field information from inside the log messages. For example...

Message=(Error) I/O error on file system 'prodops' operation WRITE inode (Message repeated 4732 times)

Message=(Error) I/O error on file system 'proxy' operation WRITE inode (Message repeated 4 times)

Message=(Error) I/O error on file system 'wwtowip' operation WRITE inode

These come from Windows event logs and I want to be able to sort on how many times these errors happen to each individual filesystem.

0 Karma
1 Solution

lguinn2
Legend

There are a number of ways to extract fields. Overview of search-time field extractions is a good resource.

For the data you have here, you could do this in the rex command as well -

yoursearchhere
| rex "(?<msg>.*?file system)\s*\'(?<file_system>.*?)\' operation (?<operation>.*)(?:\(Message repeated (?<msgcount>\d+)  times\))*"
| fillnull value=1 msgcount
| stats sum(msgcount) as MessageCount by msg file_system 

I've probably made some typo in the regular expression... but I hope you get the idea

View solution in original post

lguinn2
Legend

There are a number of ways to extract fields. Overview of search-time field extractions is a good resource.

For the data you have here, you could do this in the rex command as well -

yoursearchhere
| rex "(?<msg>.*?file system)\s*\'(?<file_system>.*?)\' operation (?<operation>.*)(?:\(Message repeated (?<msgcount>\d+)  times\))*"
| fillnull value=1 msgcount
| stats sum(msgcount) as MessageCount by msg file_system 

I've probably made some typo in the regular expression... but I hope you get the idea

mdavis43
Path Finder

Thanks, this looks great for long term usage.

0 Karma

lukejadamec
Super Champion

You should use rex to create a field from within the Message field that you can search on later.

index=main sourcetype="*security*" | rex field=Message "... error on file system '(?<filesystem>.*)' | stats count by filesystem
0 Karma

mdavis43
Path Finder

Thanks! This worked great inline.

0 Karma
Get Updates on the Splunk Community!

Optimize Cloud Monitoring

  TECH TALKS Optimize Cloud Monitoring Tuesday, August 13, 2024  |  11:00AM–12:00PM PST   Register to ...

What's New in Splunk Cloud Platform 9.2.2403?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.2.2403! Analysts can ...

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...