Splunk Dev

At least one log for two, distinct search conditions

techgeek_us
New Member

We have two jobs that run and produce a distinct log entry for each. I need to add a dashboard panel which monitors these jobs.

The success criteria is that at least one log exist for each job, so zero entries from one job and two from the other would not be a success, only >=1 for each. For this reason, I can't use an OR operator and include both conditions in the search. Oh, and the fields are not extracted.

How can I express this in a search and then in a dashboard panel which translates to something like "OK" or "2".

Thank you in advance for your time.

Tags (1)
0 Karma
1 Solution

renjith_nair
Legend

@techgeek_us ,

Easiest method is to extract job name and find status.

"your search"
|rex field=_raw "Job (?<JOB>.+) completed"|stats dc(JOB) as c|eval status=if(c>1,"OK","NOK")

Test

|makeresults|eval raw="Job one completed,Job two completed"|makemv raw delim=","|mvexpand raw|eval _raw=raw 
|rex field=_raw "Job (?<JOB>.+) completed"|stats dc(JOB) as c|eval status=if(c>1,"OK","NOK")

Another method is searching directly in the event

"your search"
|eval j1=if( searchmatch("Job one completed"),1,0)|eval j2=if( searchmatch("Job two completed"),1,0)
|stats max(j1) as j1,max(j2) as j2|eval status=if(j1==1 AND j2==1,"OK","NOK")

Test

|makeresults|eval raw="Job one completed,Job two completed"|makemv raw delim=","|mvexpand raw|eval _raw=raw 
|eval j1=if( searchmatch("Job one completed"),1,0)|eval j2=if( searchmatch("Job two completed"),1,0)
|stats max(j1) as j1,max(j2) as j2|eval status=if(j1==1 AND j2==1,"OK","NOK")
Happy Splunking!

View solution in original post

0 Karma

renjith_nair
Legend

@techgeek_us ,

Easiest method is to extract job name and find status.

"your search"
|rex field=_raw "Job (?<JOB>.+) completed"|stats dc(JOB) as c|eval status=if(c>1,"OK","NOK")

Test

|makeresults|eval raw="Job one completed,Job two completed"|makemv raw delim=","|mvexpand raw|eval _raw=raw 
|rex field=_raw "Job (?<JOB>.+) completed"|stats dc(JOB) as c|eval status=if(c>1,"OK","NOK")

Another method is searching directly in the event

"your search"
|eval j1=if( searchmatch("Job one completed"),1,0)|eval j2=if( searchmatch("Job two completed"),1,0)
|stats max(j1) as j1,max(j2) as j2|eval status=if(j1==1 AND j2==1,"OK","NOK")

Test

|makeresults|eval raw="Job one completed,Job two completed"|makemv raw delim=","|mvexpand raw|eval _raw=raw 
|eval j1=if( searchmatch("Job one completed"),1,0)|eval j2=if( searchmatch("Job two completed"),1,0)
|stats max(j1) as j1,max(j2) as j2|eval status=if(j1==1 AND j2==1,"OK","NOK")
Happy Splunking!
0 Karma

techgeek_us
New Member

Thank you so very much. I tried your first solution and was able to customize it to my needs.

0 Karma

renjith_nair
Legend

@techgeek_us ,

How do you distinguish between two jobs, are the entries different? The easiest solution would be extracting the job name and count on it. Do you have some sample events for both jobs from where we can try extracting the information?

Happy Splunking!
0 Karma

techgeek_us
New Member

The log entries would look something like:

"Job one completed"
"Job two completed"

There could be none or multiple of each. What I need is to ensure that at least one occurs for each job and then represent that on a dashboard, ideally using one query.

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...