Splunk Search

Can I add events to a transaction?

MonkeyK
Builder

I have a transaction based on a bunch of events from a common source with a common transaction ID, something like

|"search" | transaction by tid

This will get me results like

2017-04-11 04:20:32,502 tid:10001
2017-04-11 04:20:32,502 tid:10001
2017-04-11 04:20:32,502 tid:10001
2017-04-11 04:20:31,502 tid:10001
-----------------------------------------------------
2017-04-11 04:10:12,502 tid:10000
2017-04-11 04:10:12,502 tid:10000
2017-04-11 04:10:12,502 tid:10000
2017-04-11 04:10:11,502 tid:10000
-----------------------------------------------------

but I need to associate the transaction events with an event from another source. that source has an outcome that happens within 1s of the transaction in the first source.

2017-04-11 04:20:33,502 src_ip=192.168.1.99 result=success
2017-04-11 04:10:12,502 src_ip=192.168.1.15 result=fail

Is there a way to add events to my transactions or otherwise join based on the last time in the transactions? Ideally I'd like to be able to see results like:

-----------------------------------------------------
2017-04-11 04:20:33,502 src_ip=192.168.1.99 result=success
2017-04-11 04:20:32,502 tid:10001
2017-04-11 04:20:32,502 tid:10001
2017-04-11 04:20:32,502 tid:10001
2017-04-11 04:20:31,502 tid:10001
-----------------------------------------------------
2017-04-11 04:10:12,502 src_ip=192.168.1.15 result=fail
2017-04-11 04:10:12,502 tid:10000
2017-04-11 04:10:12,502 tid:10000
2017-04-11 04:10:12,502 tid:10000
2017-04-11 04:10:11,502 tid:10000
-----------------------------------------------------
0 Karma
2 Solutions

somesoni2
SplunkTrust
SplunkTrust

I guess you can try this

"search" | transaction by tid
| append ["search for source 2" ]
| transaction maxspan=1s

View solution in original post

0 Karma

woodcock
Esteemed Legend

Try this:

 "search for source 1" | transaction BY tid
| eval times=_time
| append [ "search for source 2" | eval times = (_time-1) . " " . _time . " " . (_time+1) | makemv times ]
| transaction times maxspan=1s

View solution in original post

woodcock
Esteemed Legend

Try this:

 "search for source 1" | transaction BY tid
| eval times=_time
| append [ "search for source 2" | eval times = (_time-1) . " " . _time . " " . (_time+1) | makemv times ]
| transaction times maxspan=1s

MonkeyK
Builder

Woodcock, this is a really neat idea. I think that it doesn't work for me, because it counts on times aligning to the second. To be fair, my example only showed 1s precision.

I think that somesoni2's answer is very close to what I need but that I may be running into some kind of result set limit. I think this because I get results for 1 day, but if I expand my query to a month, the 1 day results disappear.

But with some tweaking, your idea may solve a more general problem of joining datasets by time with sub-second proximity precision (which transaction does not do).
This can be important when we need to correlate datasets but there is more than 1 set of events per second.

I had been trying to solve this by using "bin _time" but the problem is that we cannot control which bin the event lands in. By adding values and comparing the additional values, I can be much more certain to land on a matchable time

for subsecond transactions on 1 source
"search for source 1" | bin _time span=200ms | eval times=_time . " " . (_time+0.2) | makemv times
| transaction times

and on two
"search for source 1" | transaction BY tid
| bin _time span=200ms | eval times=_time . " " . (_time+0.2) | makemv times
| append [ "search for source 2" | bin _time span=200ms | eval times=_time . " " . (_time+0.2) ]
| transaction times

To be fair, the bin could result in events up to 400ms apart, but this is still way better than a precision of 1s
I did run into this problem with some of my results, but it was rare enough that I figured I could just call out the discrepancy in the results

Splunk could solve the same problem by allowing smaller transaction span units

0 Karma

somesoni2
SplunkTrust
SplunkTrust

I guess you can try this

"search" | transaction by tid
| append ["search for source 2" ]
| transaction maxspan=1s
0 Karma

MonkeyK
Builder

Just tried it. The second transaction does not get combined with the first and I get all events from the appended search, so it ends up looking like

2017-04-11 04:21:33,502 src_ip=192.168.1.97 result=success


2017-04-11 04:20:36,502 src_ip=192.168.1.98 result=success


2017-04-11 04:20:33,502 src_ip=192.168.1.99 result=success


2017-04-11 04:20:32,502 tid:10001
2017-04-11 04:20:32,502 tid:10001
2017-04-11 04:20:32,502 tid:10001
2017-04-11 04:20:31,502 tid:10001


2017-04-11 04:15:02,502 src_ip=192.168.1.20 result=fail


2017-04-11 04:14:52,502 src_ip=192.168.1.19 result=fail


2017-04-11 04:13:42,502 src_ip=192.168.1.18 result=fail


2017-04-11 04:12:32,502 src_ip=192.168.1.17 result=fail


2017-04-11 04:11:22,502 src_ip=192.168.1.16 result=success


2017-04-11 04:10:12,502 src_ip=192.168.1.15 result=fail


2017-04-11 04:10:12,502 tid:10000
2017-04-11 04:10:12,502 tid:10000
2017-04-11 04:10:12,502 tid:10000
2017-04-11 04:10:11,502 tid:10000

0 Karma

MonkeyK
Builder

Whoa. I can actually almost make this work if I sort before the second transaction:
"search" | transaction by tid
| append ["search for source 2" ]
|sort 0 _time
| transaction maxspan=1s

That's what I get for over-sanitizing the question and results sets

0 Karma

MonkeyK
Builder

What I cannot figure out is why I get 1 transaction record back when I search on 3/15,
but if i search between 3/15-4/5 I get 61 transactions records back with none on 3/15.

I must be running into some some kind of limit

0 Karma

woodcock
Esteemed Legend

This is why I almost never use transaction; it has silently-enforced limits that are exceedingly easy to hit. This is one of the reasons many people say "get as much RAM as you can", even 10X of suggested amounts.

0 Karma

MonkeyK
Builder

Wow. Very useful advice! I will rethink my query and see what I can do without transactions.

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...