Splunk Search

Extract the name of the current running alert search

damucka
Builder

Hello,

I have several alerts running on minute base and would like to know within the SPL of the currently running alert what is the corresponding alert name.
My tries look as follows:

| rest /services/search/jobs 
| rex field=id "(?<jobId>[^//]*)$"
| addinfo
| where jobId = info_sid

but in the jobs, there is no way to get the name of the alert or at least i could not find it.
I tried also to scan the saved searches with the similar rest call, but there it is not possible to match it with sid. The title (alert name) is there though.
Matching with the time, like now() using the rest to saved searches I would like to avoid, it has potential to be erroneous.
Now, am I missing something?
Could you please advice?

Kind Regards,
Kamil

0 Karma

manjunathmeti
Champion

You can search scheduler logs.

index=_internal sourcetype=scheduler search_type=scheduled alert_actions!="" | table savedsearch_name, sid, app, alert_actions, scheduled_time
0 Karma

damucka
Builder

And how would I match it to the current running search? By sid?
Unfortunately I have impression that the _internal index gets written first after the execution is completed, which means I am not able to read it from the SPL of the currently running search as it does not exists yet.

0 Karma

manjunathmeti
Champion

You can get search name in /search/jobs itself field name is label.

| rest /servicesNS/-/-/search/jobs  | search dispatchState=RUNNING | table sid, label, doneProgress, dispatchState, search
0 Karma

damucka
Builder

Thank you.
After long trying the working result is following:

| appendcols [
  search index=_internal sourcetype="scheduler" suppressed=0 alert_actions!="" earliest=@d     
  | rename savedsearch_name as label 
  | rex field=sid "scheduler.+__mlbso__(?P<jobsid1>.+)_at.+"
  | convert ctime(scheduled_time) as scheduled
  | table scheduled run_time label sid jobsid1
  | addinfo
  | rex field=info_sid "scheduler.+__mlbso__(?P<jobsid2>.+)_at.+"

  | where jobsid2 != ""
  | where jobsid1 = jobsid2

  | sort by scheduled asc
  | streamstats count as triggered | fields label triggered scheduled run_time
  | sort by scheduled desc
  | eval triggering = "Alert name: " + label + "," + " # today triggered: " + triggered  + " times," + " scheduled time: " + scheduled  + "," +  " Runitme: " + run_time + " sec;" 
  | rename triggered as _triggered
  | table triggering
  | mvcombine triggering delim=","
]
| eval f=mvindex(triggering,0)
| rex mode=sed field=triggering "s/,/,    /g"
| rex mode=sed field=triggering "s/;/\n/g"
| rex field=f "Alert.+triggered: (?P<counter>.+) times.+"
| rename f as _f
| rename counter as _counter
| rename triggering as _triggering

I have one last question: in the above what is not really working is the rex with appending the tab.
I tried everything already, \x89, \t ... nothing works. I would like to replace the "," with the ",tab" to make the line a bit more readable and somehow i am not able to manage it.
Do you have perhaps idea how i could do this?

Kind Regards,
Kamil

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...