This works.  I modified the earliest / latest to match the other example to make Splunk snap to whole 10 minute buckets if I understand how that works correctly. index=anIndex source=aSource "aString" earliest=-20m@m latest=@m I am comparing results of your solution to the one provided by @bowesmana  In your solution I like that I see two results, one for each 10 minute bucket and it has a column for each 'range'  '20 to 10 Mins' & '10 to 0 Mins" I ran both queries to validate the results and both return the same results. Both are valid solutions.  How do I give 'credit' to both of you !!! ... View more

Hi, I am trying to adapt this solution to a query that I have that returns job ending times. If I am starting with this query: index=anIndex sourcetype=aSourcetype ( aJobName AND "COMPLETED OK" ) From this query I get a list of events. How can this solution be used to graph my events ? ... View more

Keeporphans=true did the trick. I saw that earlier but the way I read it, was not making sense. Thanks... ... View more

The original post said 5 Minutes which is 300 seconds. Threshold is not a calculated value and does not matter on lookback time ? Im guessing the assumptions I made are different than yours. Event data would help clarify any of those...   ... View more

Hi, I have been working on something very similar.  I am somewhat new so this might not be the 'best' solution but it will work.  Replace 300 with your threshold value.  Then when setting up your alert you can trigger an action if results are > 0. index=yourIndex sourcetype=yourSourceType ("START of JobString") earliest=-1d@d | eval startTime=_time | eval timeNow=now() | eval duration = timeNow - startTime | eval timeStringDuration = strftime(duration, "%H:%M:%S") | eval timeStringStart = strftime(startTime, "%H:%M:%S") | eval timeStringNow = strftime(timeNow, "%H:%M:%S") | where duration > 300 | table startTime, timeStringStart, timeNow, timeStringNow, duration, timeStringDuration ... View more

I got around to looking further into this today and works as advertised... Here is what I ended up with, still need to do a little date formatting but the content is 100%. index=anIndex sourcetype=aSourceType aJobName ("START of script" OR "COMPLETED OK") earliest=-7d@d   | rex "(?<event_name>(START of script)|(COMPLETED OK))" | eval event_name=CASE(event_name="START of script", "script_start", event_name="COMPLETED OK", "script_complete") | eval event_time=strftime(_time, "%Y-%m-%d %H:%M:%S") | eval aSortTime=_time | eval {event_name}_time=_time | transaction host job_name startswith=(event_name="script_start") endswith=(event_name="script_complete")   | eval "Job Name"="Fake Job Name" | eval "Host Name"=if (host="HOST1", "Host1-Name", "Host2-Name") | eval "Start Date/Time"=strftime(script_start_time, "%Y-%m-%d %H:%M:%S") | eval "Stop Date/Time"=strftime(script_complete_time, "%Y-%m-%d %H:%M:%S") | eval "Day Of Week"=strftime(script_complete_time, "%A") | eval "Duration"=tostring(duration, "duration")   | sort aSortTime | table "Host Name", "Day Of Week", "Job Name", "Start Date/Time", "Stop Date/Time", , "Duration" ... View more

I am waiting for some new log files (sourcetypes) to be added to my Splunk index. They should be available next week and will look at implementing the solution provided by  @johnhua. Since I am somewhat new to Splunk it will take me some time to understand what it is doing and then add / change the SPL to fit the formating on the output. In the meantime I am looking for a solution to my existing SPL, if there is one... ... View more

Is it possible to get the value or startTimeRaw using an index ? My thinking is set the index to 0 and then if there are more than 1 value in startTimeRaw set the index to 1 instead of 0, then get the value of startTimeRaw at that index. Original SPL: | stats range(_time) as duration values(startTimeRaw) as startTimeRaw values(endTimeRaw) as endTimeRaw by eventDate, host Pseudo Code: | eval anIndex =0 Then check the # of values in startTimeRaw and if > 0 set anIndex to the value of startTimeRaw at the last values index. ( | eval startTimeRawUsingIndex = startTimeRaw @ anIndex) | stats range(_time) as duration values(startTimeRawUsingIndex) as startTimeRaw values(endTimeRaw) as endTimeRaw by eventDate, host ... View more

I checked and for my Splunk environment mvstats is not a valid command.  I have reached out to my Splunk sys admin to see if it can be added.   In the meantime anything else you can think of that I can try ? ... View more

I tried this and it looks like its taking the last raw time for every row, not just the one with two ? index=anIndex sourcetype=aSourceType aJobName ("START of script" OR "COMPLETED OK") | eval startTimeRaw = if (match(_raw, "START of script"), _time, null()) | eval endTimeRaw = if (match(_raw, "COMPLETED OK"), _time, null()) | eventstats last(startTimeRaw) AS startTimeRaw ... View more

Interesting, I ran your SPL with the real index, sourcetype, jobname, etc... and got what appears to be some pretty good results. I will need to go back and verify the data from your SPL and my SPL, check conversions to CDT, etc... and then try to understand what exactly its doing since I am somewhat new to Splunk and have never worked with transactions before today. It will take some time to go through validation, etc..  and am still hoping there is a simple one line addition / change to my original SPL to get the latest start time for the one row that is causing me issues.   ... View more

What I am doing is looking back 8 days (earliest -8d@d latest now) and calculating the duration of job run's.  Where I am running into an issue is where a job started from one day and the previous day instead of just starting on a unique day.  Here is my whole SPL: I tried Min(...) & max(...) but it assigned one value for every row, instead of on the one row that has two results. index=anIndex sourcetype=aSourceType aJobName ("START of script" OR "COMPLETED OK") | eval startTimeRaw = if (match(_raw, "START of script"), _time, null()) | eval endTimeRaw = if (match(_raw, "COMPLETED OK"), _time, null())   | eval eventDate=strftime(_time,"%Y-%m-%d")   | stats range(_time) as duration values(startTimeRaw) as startTimeRaw values(endTimeRaw) as endTimeRaw by eventDate, host | eval durationTime = tostring (duration, "duration")   | eval aTime1= startTimeRaw- (strptime(strftime(startTimeRaw,"%Y-%m-%dT%H:%M:%S.%3N")." CDT","%Y-%m-%dT%H:%M:%S.%N%Z") - startTimeRaw) | eval startDateTimeCDT=strftime(aTime1, "%Y-%m-%d %H:%M:%S")   | eval aTime2= endTimeRaw - (strptime(strftime(endTimeRaw,"%Y-%m-%dT%H:%M:%S.%3N")." CDT","%Y-%m-%dT%H:%M:%S.%N%Z") - endTimeRaw) | eval endDateTimeCDT=strftime(aTime2, "%Y-%m-%d %H:%M:%S")   | eval startTimeCDT=strftime(aTime1, "%H:%M:%S") | eval endTimeCDT=strftime(aTime2, "%H:%M:%S") | eval host = if (host="aHost1", "aHost1-Name", "aHost2-Name") | eval "Duration"=durationTime | eval "End Time (CDT)"=endTimeCDT | eval "Start Time (CDT)"=startTimeCDT | eval "Start Date / Time (CDT)"=startDateTimeCDT | eval "End Date / Time (CDT)"=endDateTimeCDT | eval "Day Of Week"=strftime(aTime2,"%A")   | table duration, startTimeRaw, endTimeRaw, "Host Name", "Day Of Week", "Job Name", "Start Date / Time (CDT)" "End Date / Time (CDT)", "Start Time (CDT)", "End Time (CDT)", "Duration"    I have attached a screenshot of my results.  The line that is causing issues is the one with two startTimeRaw(s)... ... View more

For this example there is no unique piece of data that identifies the unique start/end time on a per day basis.  It is the same job that runs every day and its the same job name and same start and stop entries every day. Is there a way to make the date the unique identifier as I eventually want to show results by day over time? I have this working when I run for a timerange of 'today' but it does not work when I run for more than one day. i.e. 'week to date' Your example was missing 'aJobName' as I am scanning many .log files for many jobs but am currently only interested in one to get something working. index=anIndex sourcetype=aSourcetype aJobName ("START of script" OR "COMPLETED OK") | eval startTimeRaw = if (match(_raw, "START of script"), _time, null()), endTimeRaw   = if (match(_raw, "COMPLETED OK"), _time, null()) | stats range(_time) as duration values(startTimeRaw) as startTimeRaw values(endTimeRaw) as endTimeRaw by host | eval duration = tostring (duration, "duration") | table host, duration The results for a time range of today returns results like: Host Duration server1 00:17:41 When I run for week to date I get the following results: Host Duration server1 1+00:52:02 I currently only have two days of data in Splunk but I think what is happening is that each query is returning two events, which is also why in my results I only have one row instead of two (one for 9/23 and one for 9/22) but I dont know how to change the above query or approach to be able to show duration over time on a daily basis which I eventually want to dump into a timechart to show duration over time. ... View more

Two follow up questions. 1)  Even though you suggest not using append, why does it not work ?  I have a working example using appendcols and assumed append would work similar.  In the appendcols example I use: " | stats count as X " for the first query variable then " | stats count as Y " for the second variable and then use X & Y to calculate a percentage.  But for this example trying to use eval instead of stats   2) The example makes sense up to "by <some id for transaction>".  What am I trying to reference here ?  From the search I have two events and two variables (startTimeRaw & endTimeRaw). ... View more

  I am using index=anIndex cause I need to first search for a particular string to identify which servers log a  known event.  I then compare against a list of 25 known servers that are logging splunk traffic. I will see if I can use any of your suggestions in my next iteration of this. Thanks for your input ! ... View more

My first goal was to get something to work I will look at your suggestion to see if I can incorporate or change over to use this.  I have around 25 servers that im trying to monitor. Thanks for your input ! ... View more

Here is my solution: | set diff [ | search index=anIndex sourcetype1=aSourceType ("aString1" AND (host="aHostName1*" OR host="aHostName2*")) | stats count by host | table host ] [ search index=anIndex sourcetype=aSourceType2 (host="aHostName1*" OR host="aHostName2*") | stats count by host | table host ] ... View more