Group Events Based Solely on Temporal Proximity

[fervin]

Hi all,

We have a need to correlate IPS, application, and firewall logs based solely on their timestamps.

The reason is because our proxy servers and various firewalls perform Network Address Translation (NAT) and thus events don't share common src_ip or dst_ip in all cases.

I'm familiar with transactions, and they are great, but only when each event involved in the correlation shares a common field value. What we're looking for is a way to correlate based on a specified (likely very small) span of time between events of two or more sourcetypes.

Anyone have a good way of doing this? Thanks in advance,

-Frank

[Stephen_Sorkin · Accepted solution]

I would use transaction as gkanapathy suggests in his answer. You can then post-filter those transactions to only include those with BOTH sourcetypes. For example:

sourcetype=ips OR sourcetype=*syslog* | transaction maxpause=1s | search sourcetype=ips sourcetype=*syslog*

[hulahoop]

How about the search command kmeans?

http://www.splunk.com/base/Documentation/latest/SearchReference/Kmeans

[gkanapathy]

As a matter of fact transaction does not require items to share any common fields. You can just specify maxspan (time from first event) and/or maxpause (max time between events), and all events piped to it will be grouped by those parameters.

I suppose I should note that transaction is mostlyuseful for finding out and grouping events in aggregate. If you're looking at a specific set of events in a specific time range, it is much more efficient specify your time range (or use a subsearch to generate a time range around a specific event) and use the search terms to get the sourcetypes you're interested in seeing.

[fervin]

Thanks for the prompt response. So, you're right about transaction not requiring common fields, but I can't see how to force it to give me what I need for this use case. It's very likely that I'm missing something incredibly simple...

An example:

sourcetype=ips OR sourcetype=syslog | transaction maxpause=1s

This could potentially give me bundles of all syslog, with no ips events.

What I want would be the functional equivalent of something like this fake query:

sourcetype=syslog | near sourcetype=ips maxpause=1s

Does this make sense?