Splunk Search

Correlating Different Events and Calculating Time Difference

dw385
Explorer

I’m trying to report on the time difference between two related events. Both events are collected from Windows event log and are in the same index. They will occur on the same host but at different times and IDs.
Example event data:

Time: 3/11/2016 9:00:45 PM
EventID: 1000
Source: Backup Agent Message:
Backup Started
Host: Server1
Time: 3/12/2016 1:09:45 AM

EventID: 1001
Source: Backup Agent
Message: Backup Completed
Host: Server1

The second event can occur any time after the first, maybe in a few minutes or a few hours. I’m thinking I need to find event 1(1000) first and then look for the first occurrence of event 2 afterwards. The index would contain similar events from many hosts around the same time.
I’ve tried various attempts with joins, transactions, and subsearches and searching the documentation for suggestions with no luck. I’m really not concerned with any of the data in the event other than the time (which I would eval to a start and end time). The goal is to be able to run a report over a time span to show the start/end times for each host.

This gets me close but doesn’t scale to multiple hosts/time ranges:

index="wineventlog" SourceName="Backup Agent" (EventCode="1000" OR EventCode="1001") 
| transaction maxspan=4h startswith=(EventCode="1000") endswith=(EventCode="1001") 
| table host,duration
Tags (1)
0 Karma

somesoni2
Revered Legend

How many times these events can happen from one host in a day?

0 Karma

dw385
Explorer

Usually only one occurrence but its possible to see multiple in a day as jobs are restarted.

0 Karma

lguinn2
Legend

Try this first:

 index="wineventlog" SourceName="Backup Agent" (EventCode="1000" OR EventCode="1001") 
 | transaction host maxspan=4h startswith=(EventCode="1000") endswith=(EventCode="1001") 
 | table host,duration

All I did was add host to the transaction command. This should help, but it still may not be a good solution, if you have a long time range.
You could also try this:

 index="wineventlog" SourceName="Backup Agent" (EventCode="1000" OR EventCode="1001") 
| sort host,_time
| eval starttime=if(EventCode="1000",_time,null())
| eval endtime=if(EventCode="1001",_time,null())
| streamstats current=t window=2 global=f latest(starttime) as start latest(endtime) as end by host
| where isnotnull(endtime)
| eval duration=end-start

This solution should work with somewhat larger timeframes. I am not sure how well it will work if you have multiple backups for a single host.

0 Karma
Got questions? Get answers!

Join the Splunk Community Slack to learn, troubleshoot, and make connections with fellow Splunk practitioners in real time!

Meet up IRL or virtually!

Join Splunk User Groups to connect and learn in-person by region or remotely by topic or industry.

Get Updates on the Splunk Community!

Splunk Cloud Application Management in Terraform

Register   On Tuesday, August 4 at 11AM PDT / 2PM EDT, we’re diving into how you can bring Infrastructure as ...

Get Agentic with Splunk Lantern: Connect to Cisco Cloud Control, Transform ...

Splunk Lantern is Splunk’s customer success center that provides practical guidance from Splunk experts on key ...

July Community Events: Master ITSI 5.0 & Automate Splunk

Struggling with alert fatigue or feeling like you're spending more time on infrastructure maintenance than ...