Splunk Search

Best way to create a transaction of three events?

dnolan
Explorer

I've got data that looks (functionally) like this:

Event 1 contains String-A Field-X
Event 2 contains String-B Field-X Field-Y
Event 3 contains String-C Field-Y

I'm trying to correlate these three events together. I want to create a single transaction so I can calculate the duration between Event 1 and Event 2.

After some trial and error I came up with this query:

String-A OR (String-B Field-Y=*) | transaction Field-X | append [search String-C] | transaction Field-Y

That works, but it seems really ugly to me, and I'm wondering if I'm missing some better way to do this. Ideally I'd also like to be able to find examples of Event-1 with no matching Event-3, but I think I can do that with transaction as well, using keepevicted=1.

Any suggestions for a better path to my desired results?

Edit:

Ok, maybe I boiled down my example too much. Here's the original data:

Event 1:

2,51216,117,117,626,1284525367018,4172,6876,17:reqid=-1284474701,0:,0:,0,(49|A136:Msg: type = 39 BRMWroteMsg: backupid=us-pghdbp1.ariba.com_1284523604, KBytes written=3444, End of Data=1, KBytes per sec=20935.5, copy=0|)

This is a Symantec NetBackup job completing. The key pieces of Event 1 are "End of Data=1" (There were lots of records before this with that value as 0), and the backupid. I'm extracting the backupid as a field.

Event 2:

0,51216,111,111,60899,1284525448888,7992,7172,0:,203:
    ImageCopyExt_Record => {
        MasterServerName => hubmaster
        BackupID => us-pghdbp1.ariba.com_1284523604
        CopyNumber => 2
        JobID => 2698
        LifecycleSourceTag => 1
    },33:ImageCatalogImpl::updateImageCopy,1

This is Symantec scheduling the data in this backupid as part of a replication job between storage arrays. They may be multiple of these entries, for multiple backup objects all grouped together in one replication. The key elements I've identified are "ImageCopyExt_Record", the backupid (same as event 1) and the jobid (will match event 3). Other events exist with the search string and the backupid but without a jobid.

Event 3:

2,51216,116,116,5,1284526119527,6788,7860,10:jobid=2701,0:,0:,0,(14|A4:NONE|A10:SLP_HUBSLP|A3:Dup|A1:0|A50:the requested operation was successfully completed|)

This is Symantec logging that the replication completed. Here we see the matching jobid from event 2, 'A3:Dup' means this is about a replication event, and the string says completed.

So the query I had built was:

("End of Data=1" OR (ImageCopyExt_Record netbackup_jobid=*)) | transaction netbackup_backupid | append [search index=backups A3:Dup completed] | transaction netbackup_jobid 

I'm grabbing event1 and event 2 and transactioning them on the backupid, then append to that the event3 records and transactioning again. I'm sure there is a better way.

0 Karma
1 Solution

gkanapathy
Splunk Employee
Splunk Employee

Please see my comment, as I really have no idea how this is supposed to work, but I wonder if:

String-A OR String-B OR String-C | transaction Field-X Field-Y

would do what you want.

View solution in original post

gkanapathy
Splunk Employee
Splunk Employee

Please see my comment, as I really have no idea how this is supposed to work, but I wonder if:

String-A OR String-B OR String-C | transaction Field-X Field-Y

would do what you want.

gkanapathy
Splunk Employee
Splunk Employee

Yes, transaction will be transitive on each field. If a field has a value, it won't link, but if it is null, then it will be allowed to link. This behavior I think is related to the connected=t option.

0 Karma

dnolan
Explorer

I swear I tried that yesterday and it didn't give me useful results.

But I just tried it again and what I found was it found cases where event1 exist but no event2 or event3. (Either backups that aren't configured to replicate, or backups that are still replicating.) So I added to the end of that "| search completed" and it works. Somehow I didn't expect transaction to transaction on EITHER field, but on BOTH. Thanks!

0 Karma

gkanapathy
Splunk Employee
Splunk Employee

It's not clear at all to me what your transaction is supposed to do. You have abstracted away everything and provided a single example, so I find it very difficult to understand your intention, i.e., what you want it to do. I can see what your search does do, but with no way to generalize, I can't come up with something better.

0 Karma
Get Updates on the Splunk Community!

The Splunk Success Framework: Your Guide to Successful Splunk Implementations

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...

Splunk Training for All: Meet Aspiring Cybersecurity Analyst, Marc Alicea

Splunk Education believes in the value of training and certification in today’s rapidly-changing data-driven ...

Investigate Security and Threat Detection with VirusTotal and Splunk Integration

As security threats and their complexities surge, security analysts deal with increased challenges and ...