Alerting

How to determine whether a saved search was run on its cron schedule, or not?

adamsmith47
Communicator

In a report I'm building, I'm using the | map command to send emails to many recipients, each with their own custom view of data. A problem I've run into while editing the search is, I do not want to accidentally send many erroneous reports via email if I run the search while testing/editing, or even accidentally opening the search. I've come up with a rough solution, but, am wondering if someone has a better idea.

Basically I've created a macro that:
1) uses | rest to check the cron the search is scheduled for,
2) guesses at the epoch time cron_guess the search would have run at today (this logic breaks if the cron doesn't follow MM HH * * * format, e.g. 0,15,30,45 12 * * * breaks my logic)
3) checks to see if cron_guess = now()

After that, I use ranOnCron=1 to set the real email addresses, or ranOnCron=0 to set email addresses to my test account, preventing any "true" emails from going out.

This works for my purposes, but, I'd love a more robust solution if anyone knows of something. Accidentally sending hundreds of emails to hundreds of people with garbage data isn't fun.

Thanks!

[ranOnCron(3)]
args = NS_user, NS_app, saved_search
definition = eval ranOnCron=
    [| rest splunk_server=local /servicesNS/$NS_user$/$NS_app$/saved/searches
    | search title="$saved_search$"
    | rex field=cron_schedule "^(?<cron_min>\d+)\s+(?<cron_hour>\d+)\s+"
    | eval cron_guess=floor(relative_time(now(), "@d"))+tonumber(cron_min)*60+tonumber(cron_hour)*60*60
    | eval runOnCron_sec_min_hour=if(cron_guess==now(), 1, 0)
    | return $runOnCron_sec_min_hour]
0 Karma

adamsmith47
Communicator

Just re-reading my post, and to clarify what I have in mind, I'm looking for probably a | rest command and logic to determine whether the search was run by the scheduler, or if it were run manually.

Thanks again.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Why not just clone your report, disable notifications, and use that as your working copy.
Then after validation, move your changes up to the published version.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

adamsmith47
Communicator

That doesn't prevent accidents. I suppose accidents are always possible, but I'm sure you can see it'd be very easy for someone to OPEN the search, instead of click Edit to clone it... sending out hundreds of unexpected emails. Or one power user clicks on a saved search another power user created, sending out hundreds of unexpected emails.

I'm looking for a technical solution to reduce extremely visible errors, in addition to trying to be careful.

0 Karma

codebuilder
SplunkTrust
SplunkTrust

Sounds like you need an isolated development environment or more strict control of your users/roles.

----
An upvote would be appreciated and Accept Solution if it helps!
0 Karma

adamsmith47
Communicator

Yeah, life sucks.

Anything else?

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...