Alerting

Can you help me with the following error I'm getting from the following alert: "Unknown error for peer . Search Results might be incomplete."

x213217
Explorer

Hello,

I have this search that basically gets the longest current running jobs based on logs from a job scheduling system. For ones that are currently "Running", it also calculates the current runtime. This is to keep track of long running jobs.

I set this up as an alert to check for jobs running past 4 hours, then it will trigger and give a list of the jobs by email. However..the alert works 90% of the time except when an issue occurs on the server. The alert runs every hour and it is based on the past 3 days worth of logs.

This is the error "Unknown error for peer . Search Results might be incomplete. If this occurs frequently, please check on the peer." And it will trigger incomplete results. The splunk admins in our company say the search may need to be optimized, is this actually the case??

index=autosys source= jobName=
| where NOT ((LIKE(jobName, "%box%")) OR (LIKE(jobName, "%bx%")))
| stats latest(statusText) AS "latestStatus"  latest(timestamp) AS "latestTimestamp" by jobName
| where latestStatus="RUNNING"
| eval nowstring=strftime(now(), "%Y-%m-%d %H:%M:%S")
| eval now = strptime(nowstring,"%Y-%m-%d %H:%M:%S" )
| eval start = strptime(latestTimestamp, "%Y-%m-%d %H:%M:%S")
| eval diffInSecs = now-start
| eval strSecs=tostring(diffInSecs,"duration")
| eval strFormatSecs=if(match(strSecs,"\+"),replace(strSecs,"(\d+)\+(\d+)\:(\d+)\:(\d+)","\1 Day \2 Hr \3 Min \4 Secs"),replace(strSecs,"(\d+)\:(\d+)\:(\d+)","\1 Hr \2 Min \3 Secs")) 
| eval hour = diffInSecs / 3600
| sort -hour 
| where (hour > 4)
| fields jobName,latestStatus,latestTimestamp,nowstring, strFormatSecs
| rename latestStatus As "Status" latestTimestamp As "Job/Box Start Time" nowstring AS "Current Time" strFormatSecs AS "Runtime Days Hrs Min Secs"
Tags (3)
1 Solution

somesoni2
Revered Legend

It never hurts to optimize. Give this a try.

index=autosys source=* jobName=* NOT (jobName="*box*" OR jobName="*bx*")
| fields _time statusText timestamp jobName
 | stats latest(statusText) AS "latestStatus"  latest(timestamp) AS "latestTimestamp" by jobName
 | where latestStatus="RUNNING"
 | eval start = strptime(latestTimestamp, "%Y-%m-%d %H:%M:%S")
 | eval diffInSecs = now()-start
 | eval hour = diffInSecs / 3600
 | where (hour > 4)
 | eval nowstring=strftime(now(), "%Y-%m-%d %H:%M:%S")
 | sort -hour 
 | eval strSecs=tostring(diffInSecs,"duration")
 | eval strFormatSecs=if(match(strSecs,"\+"),replace(strSecs,"(\d+)\+(\d+)\:(\d+)\:(\d+)","\1 Day \2 Hr \3 Min \4 Secs"),replace(strSecs,"(\d+)\:(\d+)\:(\d+)","\1 Hr \2 Min \3 Secs")) 
 | fields jobName,latestStatus,latestTimestamp,nowstring, strFormatSecs
 | rename latestStatus As "Status" latestTimestamp As "Job/Box Start Time" nowstring AS "Current Time" strFormatSecs AS "Runtime Days Hrs Min Secs"

View solution in original post

0 Karma

somesoni2
Revered Legend

It never hurts to optimize. Give this a try.

index=autosys source=* jobName=* NOT (jobName="*box*" OR jobName="*bx*")
| fields _time statusText timestamp jobName
 | stats latest(statusText) AS "latestStatus"  latest(timestamp) AS "latestTimestamp" by jobName
 | where latestStatus="RUNNING"
 | eval start = strptime(latestTimestamp, "%Y-%m-%d %H:%M:%S")
 | eval diffInSecs = now()-start
 | eval hour = diffInSecs / 3600
 | where (hour > 4)
 | eval nowstring=strftime(now(), "%Y-%m-%d %H:%M:%S")
 | sort -hour 
 | eval strSecs=tostring(diffInSecs,"duration")
 | eval strFormatSecs=if(match(strSecs,"\+"),replace(strSecs,"(\d+)\+(\d+)\:(\d+)\:(\d+)","\1 Day \2 Hr \3 Min \4 Secs"),replace(strSecs,"(\d+)\:(\d+)\:(\d+)","\1 Hr \2 Min \3 Secs")) 
 | fields jobName,latestStatus,latestTimestamp,nowstring, strFormatSecs
 | rename latestStatus As "Status" latestTimestamp As "Job/Box Start Time" nowstring AS "Current Time" strFormatSecs AS "Runtime Days Hrs Min Secs"
0 Karma

x213217
Explorer

Thank you! I will try this as the Alert search and let you know

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

 (view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...