Splunk Search
Highlighted

Join events with similar times

Path Finder

I have an index containing failure events for both a system as a whole ("System") and individual sections of that system. When an individual section experiences a failure, two events are logged: one for the individual section and one for the system. My goal is to join the two events together (system & section) to have access to information in fields from both events. (Essentially, tag the "system" events with data from the "section" events).

About 75% of the time, the value for _time is the same for both events, which makes it easy to join them. However, sometimes the events are a few seconds apart, which means the join on time doesn't work.

How can I associate these very close (but not exactly the same time) events together?

Tags (3)
0 Karma
Highlighted

Re: Join events with similar times

Splunk Employee
Splunk Employee

Is there a common value in each event that could link them? If not, is there a way you could enrich the events so that they are associated. System mapped to section for example? If there is a common value, or you could provide a common value via a lookup file and enrichment of the event, you could use the transaction command.

Transaction has parameters around the time window for a transaction (span) as well as startswith and endswith so you can identify the events that start and end the transaction. The missing link (pun intended) would be a common value between them to identify that they are related.

This could also be a third event that somehow links the other two as well.

The transaction command will group all of the related events into one transactional event.

0 Karma
Highlighted

Re: Join events with similar times

Path Finder

Thank you for your suggestion. Unfortunately, transaction isn't appropriate for this data.
I did use a variation of finding a common value in my solution.

0 Karma
Highlighted

Re: Join events with similar times

Path Finder

I ended up coming up with a solution that takes care of the majority of cases where I want to join on time but the times aren't identical. It's a bit of a kludge, but it works most of the time.

Turns out, most of the events that don't match perfectly are within a few milliseconds of each other. I decided to create a field called joinTime where I take the value of _time and reformat it without the milliseconds ( "%F %T"). Thus, I can perform my join using the joinTime field, which is now the same for the vast majority of events that need to be combined.

I'm still on the hunt for something better, but this will work in the meantime.

View solution in original post

0 Karma