Splunk Search

How to grab a count of logs over a field while excluding specific logs.

Splunkster45
Communicator

I have a search query that has a field called "message_text" that I run a stats command, counting the number of log entries per message_text.

index=main env=prod  | stats count(message_text) as count | where count > 100000

For one particular message_text, I would like to exclude a specific subset of logs. I have a built a rex command that allows me to specifically query these log files.

index=main env=prod  message_text="Failed to*" | rex field=_raw "INSERT  INTO (?<model>\w+\.\w+).*"| search model="MODEL_1" 

However, I'm not sure how to combine the two ideas. I want to grab a count of all logs by message_text while excluding logs for a specific message_text that match a rex command. Below is what I thought would work, but it doesn't.

index=main env=prod | rex field=_raw "INSERT  INTO (?<model>\w+\.\w+).*"| where model!="MODEL_1" | stats count(message_text) as count | where count > 100000

I feel that when I add the rex command, followed by the where clause, my search is too granular. Now instead of looking at all message_texts, I'm feel that the above query would only be looking at logs that have a field call model whose value doesn't equal the specified model.

Does anyone have any thoughts on the matter?

Splunkster45

Tags (2)
0 Karma
1 Solution

woodcock
Esteemed Legend

Try this:

 index=main env=prod  message_text="Failed to*" | rex "INSERT  INTO (?<model>\w+\.\w+).*"| search NOT model="MODEL_1" | stats count(message_text) as count | where count > 100000

The problem is that NOT model="MODEL_1" is not the same as model!="MODEL_1". The former is the exact opposite of model="MODEL_1", which is what you desire; the latter is similar but drops all events that do not have a (non-null) field named model (i.e. field model exists AND does not have value MODEL_1), which is why you were losing events unexpectedly.

View solution in original post

woodcock
Esteemed Legend

Try this:

 index=main env=prod  message_text="Failed to*" | rex "INSERT  INTO (?<model>\w+\.\w+).*"| search NOT model="MODEL_1" | stats count(message_text) as count | where count > 100000

The problem is that NOT model="MODEL_1" is not the same as model!="MODEL_1". The former is the exact opposite of model="MODEL_1", which is what you desire; the latter is similar but drops all events that do not have a (non-null) field named model (i.e. field model exists AND does not have value MODEL_1), which is why you were losing events unexpectedly.

Splunkster45
Communicator

Thanks, this works great! I've never seen the explanation of != and Not, but I've seen that they are different. Thanks for the explanation.

0 Karma
Get Updates on the Splunk Community!

Splunk Observability Cloud's AI Assistant in Action Series: Auditing Compliance and ...

This is the third post in the Splunk Observability Cloud’s AI Assistant in Action series that digs into how to ...

Splunk Community Badges!

  Hey everyone! Ready to earn some serious bragging rights in the community? Along with our existing badges ...

What You Read The Most: Splunk Lantern’s Most Popular Articles!

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...