topic Re: Splunk alert based on condition match in Splunk Search

Splunk alert based on condition match

Vch — Mon, 05 Feb 2024 20:02:53 GMT

Hi,

I have two splunk search -1, search-2

i have to create splunk alert for search-2 based on search-1. If search-1 count greater than 0 then trigger search-2 alert

regards

vch

Re: Splunk alert based on condition match

Richfez — Mon, 05 Feb 2024 20:20:23 GMT

My first reasonable thought is that you just need to rewrite the two searches and combine them into one.

We can help with this! What do you have for search 1 and search 2 right now?

(Don't forget to use the <code> button to paste searches, and if you have to obfuscate a bit of it, feel free - but try to keep the same structure to the searches!)

Re: Splunk alert based on condition match

Vch — Mon, 05 Feb 2024 21:03:54 GMT

search1: index="*" sourcetype="*" "Generating Event gz File for*"

search2: index="*" sourcetype="*" "File Processed*"

if search1 results greater than 0 then only search2 alert should trigger email alert.

Re: Splunk alert based on condition match

Richfez — Mon, 05 Feb 2024 22:32:13 GMT

Great. Those two searches should be able to be easily combined into one.

Unfortunately, I've thought about this and I'm not sure I have quite enough information yet because I feel there's a *lot* still left unsaid.

So it would be great if you could describe the use case in a little more detail just using words and English, ignoring how you think the Splunk solution will be formulated.

I'm guessing something like - "whenever a new gz file is created, we need to check if that file was also processed or not and send an email with that information as an alert." That leaves as open questions

how long is the time period involved
how often will you have this alert scheduled for (different from the first question!)
is it a 1 to 1 relationship between "create" events and and "processing" events
what's the maximum time difference between those two events
does it matter more if a file gets created but not processed, or does that situation matter less, or is this actually the only thing that matters
do you already have the filename being extracted as a field in these two events
how often do you expect the pair of messages (daily? hourly? hundreds per second?)

The reason for so many questions is that there are quite a few ways to approach this, some may be better in certain circumstances, some may be better in others.

All in all, the details matter, but I'm sure if we get good answers to those (and perhaps a sample of the two events too) that we'll get you on your way soon.

Re: Splunk alert based on condition match

Vch — Tue, 06 Feb 2024 11:29:09 GMT

Sorry for not being more descriptive, both searches has different indexes.

i want to alert when search1 AND search2 result greater than zero.

how long is the time period involved- only one time in a day.
how often will you have this alert scheduled for (different from the first question!) - first and second searches can be done at same time, because right after few seconds of file received file will be processed
is it a 1 to 1 relationship between "create" events and and "processing" events - yes
what's the maximum time difference between those two events - maximum 1 hr 1 minute
does it matter more if a file gets created but not processed, or does that situation matter less, or is this actually the only thing that matters - yes its critical if file received( search1) and not processed (search2)
do you already have the filename being extracted as a field in these two events - yes i have
how often do you expect the pair of messages (daily? hourly? hundreds per second?) - daily once

Re: Splunk alert based on condition match

Richfez — Tue, 06 Feb 2024 13:27:27 GMT

Oh lovely, the "once per day" does wonders for simplifying the problem's edges. 🙂

So there's a few different ways to handle this then. Let's go through some options. I think our base search will be something like

(index="a" sourcetype="x" "Generating Event gz File for*") OR (index="b" sourcetype="y" "File Processed*")

I'm giving you the search piece by piece, with the idea you'll paste each piece in, see what the results are (perhaps with something like `| table *` after it), so you understand what it's doing before you add the next piece. (Note some are "add this to the end" and others are "replace the last one with this one", so just be aware)

Anyway, that's what many of us call a 'data salad'. Splunk handles messy stuff just fine. Toss it all in the salad, then later we'll add croutons and dressing. That should give you all the data - both sides of it.

Now, from here you could do something as simple as counting the results. Add this to the end.

| stats count

If all is well you will have an answer of 2. If the process is broken you may get 1, and if it's not run yet today you'll get 0. This could be used as is, but I feel it's rather plain and the alert will be sort of dumb and uninteresting and without context.

The dumb way to make it interesting at the end is eval the count so it says words. Add this to the end:

| eval status = case(if(count==2), "Everything processed correctly.", if(count==1, "Danger Will Robinson, it didn't process right!", true(), "I don't know what's going on, nothing came in today at all!")

Now when you run it, you'll get some words that would possibly be useful in the alert!

But this is still just kind of "not using the information we have available"

So, replacing the entire | stats ... through the end with this new stats + stuff (eg after the base search at the top):

| eval generated = if(searchmatch("Generating Event gz File for"), 1,0) | eval processed = if(searchmatch("File Processed"), 1,0) | stats count(generated) AS generated, count(processed) AS processed BY filename | eval status = case(generated == 1 AND processed == 1, "Received and Processed " . Filename, generated == 1 AND processed = 0, "NOT PROCESSED " . filename, true(), "Nothing reported at all")

What that does is, before we stats we create some fields (generated and processed) with a 0 or 1 in them (e.g. false or true). We group those by filename (just in case!) with the stats, then create a "status" field that's got some information plus the filename.

It should work? I mean, I don't have your data but at least it generates no errors. Feel free to break it down - start by adding the two evals to see that THEY work right, then add the stats to see if it counts right, etc...

Let me know what else this might need to do! We could include a time so that you could run historical reports... there's all sorts of other things you could do with it.

Re: Splunk alert based on condition match

Vch — Wed, 07 Feb 2024 23:19:35 GMT

Thank you so much. This is really awesome .