Splunk Search

Hung Job Monitoring via Logs

Mick_OBrien
Path Finder

I have a search string that details the last log entry for all running jobs [shown in ascending order] bar a few jobs [as either completed OR of little interest]...

index=ee_rpa_uipath_platform AND OrganizationUnitID=19 | dedup RobotName | table User, RobotName, _time, Message | sort _time | search (NOT Message IN("*complete*", "*ended*") AND NOT RobotName="ALWORKER*")

What I now want is to reduce the returned set to logs that are over 30 mins old i.e. if the log file has NOT been updated in the past 30mins then there is a good chance that the job has hung

I have tried the following command...

index=ee_rpa_uipath_platform AND OrganizationUnitID=19 | dedup RobotName | table User, RobotName, _time, Message | sort _time | search (NOT Message IN("*complete*", "*ended*") AND NOT RobotName="ALWORKER*") | where earliest=-30m@m

...but I receive the following error...

"Error in 'where' command: The operator at 'm@m' is invalid."

Two questions

  1. What's wrong with my syntax and/or
  2. Is there a better way to to get the oldest last log entry from a set of job log files
Labels (1)
0 Karma

Mick_OBrien
Path Finder

Re-visited, re-thought and re-wrote.  This seems to work...

index=ee_rpa_uipath_platform AND OrganizationUnitID=19 | dedup RobotName | table User, RobotName, _time, Message | sort _time | search (NOT Message IN("*complete*", "*ended*") AND NOT RobotName="ALWORKER*") | where _time > relative_time(now(), "-1440m@m") AND _time < relative_time(now(), "-30m@m")

 I still do not understand what was wrong with original search string syntax

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...

New in Observability Cloud - Explicit Bucket Histograms

Splunk introduces native support for histograms as a metric data type within Observability Cloud with Explicit ...