Alerting

Why is my search in my alert returning 0 events when the same search is returning more than 1 event when ran manually

loveforsplunk
Explorer

I have the below Query:

index=index host=host source=source keyword earliest = -24h@h latest = now | join [search index=index host=host keyword2 source=source | stats count(eval(searchmatch("keyword2"))) AS SearchIfZero] | where SearchIfZero>0 and keyword <1 | stats count(keyword) as Result by keyword

My alert condition is : search Result > 0

earliest Time = -24h@h
latest = now

I have kept the check every two minutes so as to test if I am getting any alert.
But unfortunately , I am getting 0 events in the alert section while I am getting 4 events when searched manually.

I am trying to figure it out whole day. Any help will be highly appreciated!!!

Tags (1)
0 Karma
1 Solution

DalJeanis
SplunkTrust
SplunkTrust

Okay, this looks like it's a different attempt to do the same thing you were having trouble with in this one -

https://answers.splunk.com/answers/544701/getting-error-error-in-eval-command-the-expression.html

The answer is pretty much the same...along with the same advice: be sure to tell us the PURPOSE of the thing you are trying to debug. You can get your desired alert with the code I've posted over there, so the only thing left to do here is to review your code for anything that you need to learn about.

So, here's all the errors I could detect in a few minutes of looking at THIS version of the code

1) When using a join, at your level of experience, ALWAYS specify the exact fields you are joining on. The way you have coded it, it will attempt to join the left and right sides on any fields that happen to be present in both sides. The ONLY field present in the right side is SearchIfZero, which is NEVER present in the left, so you will never get any results out of the join. That's the answer to the question you asked above.

2) I don't believe splunk will let you stats count(keyword) when the stats is by keyword. However, since keyword must be present on any record that survives stats when you count by keyword, you could just use count, if that's what you were trying to do.

View solution in original post

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Okay, this looks like it's a different attempt to do the same thing you were having trouble with in this one -

https://answers.splunk.com/answers/544701/getting-error-error-in-eval-command-the-expression.html

The answer is pretty much the same...along with the same advice: be sure to tell us the PURPOSE of the thing you are trying to debug. You can get your desired alert with the code I've posted over there, so the only thing left to do here is to review your code for anything that you need to learn about.

So, here's all the errors I could detect in a few minutes of looking at THIS version of the code

1) When using a join, at your level of experience, ALWAYS specify the exact fields you are joining on. The way you have coded it, it will attempt to join the left and right sides on any fields that happen to be present in both sides. The ONLY field present in the right side is SearchIfZero, which is NEVER present in the left, so you will never get any results out of the join. That's the answer to the question you asked above.

2) I don't believe splunk will let you stats count(keyword) when the stats is by keyword. However, since keyword must be present on any record that survives stats when you count by keyword, you could just use count, if that's what you were trying to do.

0 Karma

loveforsplunk
Explorer

okay .. let me explain you the purpose of the thing.. there is one log file containing two keywords say k1 and k2. My intention is if k2 is present (which is my subsearch here where I am trying to see if k2 count is not zero) then only it will search for k1 and will return the results of k1 only if k1 value are less than 1.
I hope I could make u clear about it.

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

Yes, that's what I figured.

Don't do that, it's very inefficient and overcomplicated in splunk. Any way that you code a subsearch or "conditional" search, you end up generating lots of unnecessary overhead, and it almost never works the way you'd hope or expect. Aim for a single elegant search that collects everything you might need in a single pass, and then drops whatever you don't need as soon as you are sure you don't need it.

Just do both searches and combine the results, then decide what parts of the result you need to use, and what you can ignore, like I did for you in the other question.

0 Karma

loveforsplunk
Explorer

Yes Sir, Thanks your query worked perfect for me . I have one question while I was doing some research on your query:

earliest=-5m index=foo host=bar source=baz ("keyword" OR "keyword1")
| stats count(eval(searchmatch("keyword0")) as k0, max(keyword1) as k1
| fillnull value=0 k0 k1
| where k0=0 and k1<5

In my original events , k1 was already extracted and I get to see its proper values under fields. But when I tried to modify your search and do something like this , it did not work and I am wondering why ??

earliest=-5m index=foo host=bar source=baz ("keyword" OR "keyword1")
| stats count(eval(searchmatch("keyword0")) as k0
| where k0=0 and k1<5

Shouldn't it show any result under stats tab??

0 Karma

DalJeanis
SplunkTrust
SplunkTrust

@loveforsplunk - in the later search, after you run this command...

 | stats count(something) as k0 

... field k1 does not exist.

0 Karma

micahkemp
Champion

How long does that subsearch take to run, if you run it by itself?

Your join subsearch looks like it returns just one field, SearchIfZero, and your join statement looks like it's trying to join on the field searchresult. Is this the actual search you're running, or has it been modified to post here?

0 Karma

loveforsplunk
Explorer

oh Sorry .. I had just kept join .. its a typo here I corrected in my question.

When I run my subsearch only, it takes around 5 secs.

0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...