topic Re: How to display procedures that don't have events as failures? in Splunk Search

How to display procedures that don't have events as failures?

SteveChai427 — Fri, 08 Jan 2021 13:47:57 GMT

Hello good people of the splunk community. I'm fairly new to splunk so sorry if this is a newb question.

I have a search that retrieves only events with certain field values in the Procedure_Name or Process_Name fields, groups them by our scheduling cycle, and displays which procedures/processes failed (indicated by activity code not being 2000):

(index=app host=myhost sourcetype=mysourcetype) OR (index=myindex source=mysource) earliest=-1w@w latest=now | where Process_Name IN ("Process1","Process2","Process3"..."Process26") OR Procedure_Name IN ("Procedure1","Procedure2","Procedure3"..."Procedure26")) | fields Procedure_Name,Process_Name,Activity_Code, UpdatedDate | eval Procedure_Name=coalesce(Process_Name, Procedure_Name) | eval update = strptime( UpdatedDate, "%Y-%m-%d %H:%M:%S") | eval Day = relative_time(update,"@d") - if((tonumber(strftime(update, "%H%M")) < 1400), (24*60*60), 0) | dedup Procedure_Name Day | stats count(eval(Activity_Code = "2000")) as Success_Count, values(eval(if(Activity_Code !="2000", Procedure_Name,null()))) as Failures, values(Procedure_Name) as AllProcedures, values(UpdatedDate) as UpdatedDate, count as Procedure_Count by Day | eval Success_Percent = round(((Success_Count/Procedure_Count)*100),2) | sort - Day | eval Day = strftime(Day, "%F") | table Day, Success_Count, Procedure_Count, Success_Percent, Failures, AllProcedures,UpdatedDate

The process and procedure lists I'm checking for are actually identical, so Process1 is the same as Procedure1, Process6=Procedure6, etc.

However I want to account for procedures/processes that failed to run at all since we consider that a failure too. But because they didn't run there are no events for them. Is there some way to compare my list of procedures/processes that should be there to the list that's actually there(AllProcedures) and add the difference to my failures list or another list like "FailedToRun"?

Re: How to display procedures that don't have events as failures?

richgalloway — Fri, 08 Jan 2021 15:14:53 GMT

Finding something that is not there is not Splunk's strong suit. See this blog entry for a good write-up on it.

https://www.duanewaddle.com/proving-a-negative/

Re: How to display procedures that don't have events as failures?

SteveChai427 — Fri, 08 Jan 2021 15:40:43 GMT

I'll try what's in that article and see if that does the trick. I had a feeling I'd need to do a lookup table but I don't have permissions to make one so it's a little cumbersome.

Re: How to display procedures that don't have events as failures?

schilds427 — Thu, 14 Jan 2021 21:07:55 GMT

Hello, I'm the same person but I had to get a new account. My lookup table is up and I am able to pull data from it. It's arranged like this:

1.0 Procedures	2.0_A_Procedures	2.0_B_PRocedures	3.0_Procedures
various procedures	various procedures	various procedures	various procedures

My different searches are only concerned with one column each. So the search above is only concerned with the 1.0_Procedures column but other searches use the other columns. I want to add the events that are absent from splunk but present in the lookup table to the failures field. So something like:

values(eval(if(Activity_Code !="2000" OR Procedure_Name NOT [|inputlookup chubDashboardProcedures.csv | fields 1.0_Procedures | rename 1.0_Procedures as search|format "" "(" "" ")" "OR" ""], Procedure_Name,null()))) as Failures

Except I get a quote/parenthesis mismatch error with that. Is there any way to do that?

Re: How to display procedures that don't have events as failures?

richgalloway — Fri, 15 Jan 2021 15:43:04 GMT

Try using the format command without arguments - the extra parentheses shouldn't affect the results. If that doesn't work, try " " instead of "".