Oh lovely, the "once per day" does wonders for simplifying the problem's edges. 🙂 So there's a few different ways to handle this then. Let's go through some options. I think our base search will be something like (index="a" sourcetype="x" "Generating Event gz File for*") OR (index="b" sourcetype="y" "File Processed*") I'm giving you the search piece by piece, with the idea you'll paste each piece in, see what the results are (perhaps with something like `| table *` after it), so you understand what it's doing before you add the next piece. (Note some are "add this to the end" and others are "replace the last one with this one", so just be aware) Anyway, that's what many of us call a 'data salad'. Splunk handles messy stuff just fine. Toss it all in the salad, then later we'll add croutons and dressing. That should give you all the data - both sides of it. Now, from here you could do something as simple as counting the results. Add this to the end. | stats count If all is well you will have an answer of 2. If the process is broken you may get 1, and if it's not run yet today you'll get 0. This could be used as is, but I feel it's rather plain and the alert will be sort of dumb and uninteresting and without context. The dumb way to make it interesting at the end is eval the count so it says words. Add this to the end: | eval status = case(if(count==2), "Everything processed correctly.",
if(count==1, "Danger Will Robinson, it didn't process right!",
true(), "I don't know what's going on, nothing came in today at all!") Now when you run it, you'll get some words that would possibly be useful in the alert! But this is still just kind of "not using the information we have available" So, replacing the entire | stats ... through the end with this new stats + stuff (eg after the base search at the top): | eval generated = if(searchmatch("Generating Event gz File for"), 1,0)
| eval processed = if(searchmatch("File Processed"), 1,0)
| stats count(generated) AS generated, count(processed) AS processed BY filename
| eval status = case(generated == 1 AND processed == 1, "Received and Processed " . Filename,
generated == 1 AND processed = 0, "NOT PROCESSED " . filename,
true(), "Nothing reported at all") What that does is, before we stats we create some fields (generated and processed) with a 0 or 1 in them (e.g. false or true). We group those by filename (just in case!) with the stats, then create a "status" field that's got some information plus the filename. It should work? I mean, I don't have your data but at least it generates no errors. Feel free to break it down - start by adding the two evals to see that THEY work right, then add the stats to see if it counts right, etc... Let me know what else this might need to do! We could include a time so that you could run historical reports... there's all sorts of other things you could do with it.
... View more