topic Re: Multiple error stements with retry in Splunk Search

Multiple error stements with retry

markthompson — Fri, 02 Jan 2015 08:58:00 GMT

Hello,
Well we have a job that runs and produces log files that runs and if it fails, it retries up to 3x.

How would we get splunk to take the latest value for that day of the log file so we can detect whether or not it has failed overall or not.

Thanks

Re: Multiple error stements with retry

martin_mueller — Fri, 02 Jan 2015 15:37:54 GMT

What does a failure look like in the logs? What does a successful run look like?

Re: Multiple error stements with retry

markthompson — Mon, 05 Jan 2015 08:23:50 GMT

Hey Martin,
it will be {Timestamp}: Error - {Error type}
success will be something like The run has been completed successfully.

Re: Multiple error stements with retry

martin_mueller — Mon, 05 Jan 2015 17:08:20 GMT

Try this:

index=foo sourcetype=bar "the run has been completed successfully" | timechart span=1d count | where count=0

That should give you days without a successful run.

Re: Multiple error stements with retry

markthompson — Tue, 06 Jan 2015 08:30:14 GMT

Hi Martin, thats not particularly what we're looking for, we are using rangemaps to display traffic lights of the current status, and if it's warning but is succeeding after retry it should go amber, if its failing altogether, red, and if successful then green.

Re: Multiple error stements with retry

markthompson — Wed, 07 Jan 2015 15:18:41 GMT

I have solved this using the eventstats command, example below:
index="{INDEX}" host={HOST} |
transaction startswith="{STARTSWITH}" endswith="{ENDSWITH}" |
eval StartTime=_time |
eventstats max(StartTime) as max |
where max=StartTime

This will select the latest event only.