Alerting

Why is my search in my alert returning 0 events when the same search is returning more than 1 event when ran manually

loveforsplunk
Explorer

I have the below Query:

index=index host=host source=source keyword earliest = -24h@h latest = now | join [search index=index host=host keyword2 source=source | stats count(eval(searchmatch("keyword2"))) AS SearchIfZero] | where SearchIfZero>0 and keyword <1 | stats count(keyword) as Result by keyword

My alert condition is : search Result > 0

earliest Time = -24h@h
latest = now

I have kept the check every two minutes so as to test if I am getting any alert.
But unfortunately , I am getting 0 events in the alert section while I am getting 4 events when searched manually.

I am trying to figure it out whole day. Any help will be highly appreciated!!!

Tags (1)
0 Karma
1 Solution

DalJeanis
Legend

Okay, this looks like it's a different attempt to do the same thing you were having trouble with in this one -

https://answers.splunk.com/answers/544701/getting-error-error-in-eval-command-the-expression.html

The answer is pretty much the same...along with the same advice: be sure to tell us the PURPOSE of the thing you are trying to debug. You can get your desired alert with the code I've posted over there, so the only thing left to do here is to review your code for anything that you need to learn about.

So, here's all the errors I could detect in a few minutes of looking at THIS version of the code

1) When using a join, at your level of experience, ALWAYS specify the exact fields you are joining on. The way you have coded it, it will attempt to join the left and right sides on any fields that happen to be present in both sides. The ONLY field present in the right side is SearchIfZero, which is NEVER present in the left, so you will never get any results out of the join. That's the answer to the question you asked above.

2) I don't believe splunk will let you stats count(keyword) when the stats is by keyword. However, since keyword must be present on any record that survives stats when you count by keyword, you could just use count, if that's what you were trying to do.

View solution in original post

0 Karma

DalJeanis
Legend

Okay, this looks like it's a different attempt to do the same thing you were having trouble with in this one -

https://answers.splunk.com/answers/544701/getting-error-error-in-eval-command-the-expression.html

The answer is pretty much the same...along with the same advice: be sure to tell us the PURPOSE of the thing you are trying to debug. You can get your desired alert with the code I've posted over there, so the only thing left to do here is to review your code for anything that you need to learn about.

So, here's all the errors I could detect in a few minutes of looking at THIS version of the code

1) When using a join, at your level of experience, ALWAYS specify the exact fields you are joining on. The way you have coded it, it will attempt to join the left and right sides on any fields that happen to be present in both sides. The ONLY field present in the right side is SearchIfZero, which is NEVER present in the left, so you will never get any results out of the join. That's the answer to the question you asked above.

2) I don't believe splunk will let you stats count(keyword) when the stats is by keyword. However, since keyword must be present on any record that survives stats when you count by keyword, you could just use count, if that's what you were trying to do.

0 Karma

loveforsplunk
Explorer

okay .. let me explain you the purpose of the thing.. there is one log file containing two keywords say k1 and k2. My intention is if k2 is present (which is my subsearch here where I am trying to see if k2 count is not zero) then only it will search for k1 and will return the results of k1 only if k1 value are less than 1.
I hope I could make u clear about it.

0 Karma

DalJeanis
Legend

Yes, that's what I figured.

Don't do that, it's very inefficient and overcomplicated in splunk. Any way that you code a subsearch or "conditional" search, you end up generating lots of unnecessary overhead, and it almost never works the way you'd hope or expect. Aim for a single elegant search that collects everything you might need in a single pass, and then drops whatever you don't need as soon as you are sure you don't need it.

Just do both searches and combine the results, then decide what parts of the result you need to use, and what you can ignore, like I did for you in the other question.

0 Karma

loveforsplunk
Explorer

Yes Sir, Thanks your query worked perfect for me . I have one question while I was doing some research on your query:

earliest=-5m index=foo host=bar source=baz ("keyword" OR "keyword1")
| stats count(eval(searchmatch("keyword0")) as k0, max(keyword1) as k1
| fillnull value=0 k0 k1
| where k0=0 and k1<5

In my original events , k1 was already extracted and I get to see its proper values under fields. But when I tried to modify your search and do something like this , it did not work and I am wondering why ??

earliest=-5m index=foo host=bar source=baz ("keyword" OR "keyword1")
| stats count(eval(searchmatch("keyword0")) as k0
| where k0=0 and k1<5

Shouldn't it show any result under stats tab??

0 Karma

DalJeanis
Legend

@loveforsplunk - in the later search, after you run this command...

 | stats count(something) as k0 

... field k1 does not exist.

0 Karma

micahkemp
Champion

How long does that subsearch take to run, if you run it by itself?

Your join subsearch looks like it returns just one field, SearchIfZero, and your join statement looks like it's trying to join on the field searchresult. Is this the actual search you're running, or has it been modified to post here?

0 Karma

loveforsplunk
Explorer

oh Sorry .. I had just kept join .. its a typo here I corrected in my question.

When I run my subsearch only, it takes around 5 secs.

0 Karma
Get Updates on the Splunk Community!

Introducing the Splunk Community Dashboard Challenge!

Welcome to Splunk Community Dashboard Challenge! This is your chance to showcase your skills in creating ...

Get the T-shirt to Prove You Survived Splunk University Bootcamp

As if Splunk University, in Las Vegas, in-person, with three days of bootcamps and labs weren’t enough, now ...

Wondering How to Build Resiliency in the Cloud?

IT leaders are choosing Splunk Cloud as an ideal cloud transformation platform to drive business resilience,  ...