Can the transaction command be used to group events where different fields have the same value?

[hulahoop]

Use Case: Correlate logon events from a Windows desktop to events on the domain controller.

Sample (shortened) event from the desktop:

CEF:0|Microsoft|Microsoft Windows||Security:528|Successful Logon|Low| eventId=9484152 externalId=528 msg=RemoteInteractive: A user logged on to this computer remotely using Terminal Services or a Remote Desktop connection. art=1261605081785 src=10.150.28.43 suser=svc_eiq duser=svc_eiq cn1=10 cn1Label=LogonType dvc=10.151.113.33

Sample (shortened) events from the domain controller:

CEF:0|Microsoft|Microsoft Windows||Security:540|Successful Network Logon|Low| eventId=110125027 externalId=540 msg=Network: A user or computer logged on to this computer from the network. art=1261604956463 src=10.151.113.33 suser=- duser=svc_eiq cn1=3 cn1Label=LogonType dvc=10.151.118.38

CEF:0|Microsoft|Microsoft Windows||Security:540|Successful Network Logon|Low| eventId=110125025 externalId=540 msg=Network: A user or computer logged on to this computer from the network. art=1261604956463 src=10.151.113.33 suser=- duser=svc_eiq cn1=3 cn1Label=LogonType dvc=10.151.118.38

CEF:0|Microsoft|Microsoft Windows||Security:540|Successful Network Logon|Low| eventId=110124994 externalId=540 msg=Network: A user or computer logged on to this computer from the network. art=1261604956197 src=10.151.113.33 suser=- duser=svc_eiq cn1=3 cn1Label=LogonType dvc=10.151.118.38

CEF:0|Microsoft|Microsoft Windows||Security:540|Successful Network Logon|Low| eventId=110124964 externalId=540 msg=Network: A user or computer logged on to this computer from the network. art=1261604955991 src=10.151.113.33 suser=- duser=svc_eiq cn1=3 cn1Label=LogonType dvc=10.151.118.38

The events on the domain controller occur within 1 second of the logon event being generated on the endpoint. The events on the DC and endpoint are linked by the dvc field on the endpoint and the src field on the DC. The goal is to present linked events occurring within 1 minute of each other as a single transaction.

Proposed Splunk transaction search:

source=*event*.log (suser=svc_eiq OR duser=svc_eiq) (externalId=528 OR externalId=540) (cn1=10 OR cn1=2 OR cn1=3) | transaction dvc src maxspan=1m maxpause=3s

The result is 2 events/transactions instead of 1. The transaction command first groups all events with the same dvc value, then events with the same src value.

How do I get a transaction based on the same value of both dvc and src? Is it possible to accomplish this with the transaction command?

An alternate approach we've tried is use a subsearch. The inner search first finds the events of interest on the desktop then passes the dvc field to the outer search renamed as the src field. The complete search will present the relevant domain controller events. The difficulty with this approach is with introducing the time dimension--events occurring within 1 minute of each other. It's not clear to me how to pass time as the art field from the inner to outer search without affecting the search criteria of the outer search. We want to do something like this (but it doesn't work):

sourcetype=cef externalId=540 cn1=3 [search sourcetype=cef suser=svc_* externalId=528 (cn1=10 OR cn1=2) | top dvc by suser | fields + dvc,suser,art | rename dvc as src | rename suser as duser | rename art as start_art] | eval delta_art=start_art-art | where delta_art<1m

[hulahoop]

Yes, Gerald and David's suggestion does indeed work in 4.0 and 4.1. The resulting search to perform the correlation is:

sourcetype=cef (suser=svc_eiq OR duser=svc_eiq) (externalId=528 OR externalId=540) (cn1=10 OR cn1=2 OR cn1=3) | eval x=if(externalId=540,src,dvc) | transaction x maxspan=1m maxpause=3s

This is presuming events with ID 540 only happen on the domain controller and not on the desktops. Otherwise, another way to distinguish between desktop and DC events needs to be used.

Thank you for the pointers!

[carasso · Accepted solution]

if I understand this correctly, before the transaction command, determine which field you want to use based on the source.

... | eval unifyingField = if(source=dc,src, dvc) ...

(here i'm assuming the source of dc events is called dc)

In Splunk 4.1 transactions support field unification with multi-valued fields, so you can accomplish your goal by having a multivalued field (e.g. unifyingField) with both the src and dvc values and if any value is shared between events, they are compatible and can be in the same transaction.

[hulahoop]

Thank you, Stephen! I didn't see your post before posting my answer. 🙂

[Stephen_Sorkin]

Yes, using eval to generate the unifying field will work in 4.0. The easiest way is: source=event.log (suser=svc_eiq OR duser=svc_eiq) (externalId=528 OR externalId=540) (cn1=10 OR cn1=2 OR cn1=3) | eval unifyingField = if(externalId==540, src, dvc) | transaction unifyingField maxspan=1m maxpause=3s – Stephen Sorkin♦ 0 secs ago

[hulahoop]

Thank you, David... but I'm not quite following. Is it possible to do this in 4.0? If so, can you kindly elaborate on how to identify the unifying field and then use it when calling the transaction? Unfortunately, both the desktop events and domain controller events are collected by Arcsight logger and squashed into CEF format. They share the same source(=tcp:5140) and sourcetype(=cef).

[gkanapathy]

I would combine the two fields into a single multivalued field:

... | eval x=dvc+","+src | makemv delim="," x | transaction x

Update:

Yeah, it won't work in 4.0. You just need to do what David said.

... | eval x=if(source=DC,src,dvc) | transaction x

Presumably there is some way you can tell the difference between the DC and the device. You could you any eval expression, or do a lookup first on a list of DCs or whatever.

[hulahoop]

Are you saying it will work in 4.1?

[gkanapathy]

Alright fine it won't work in 4.0

[hulahoop]

Gerald, this doesn't work. 😞 It produces the same result as

| transaction dvc src maxspan=1m maxpause=3s