topic Re: Group Events Based Solely on Temporal Proximity in Splunk Search

Group Events Based Solely on Temporal Proximity

fervin — Fri, 03 Sep 2010 00:01:59 GMT

Hi all,

We have a need to correlate IPS, application, and firewall logs based solely on their timestamps.

The reason is because our proxy servers and various firewalls perform Network Address Translation (NAT) and thus events don't share common src_ip or dst_ip in all cases.

I'm familiar with transactions, and they are great, but only when each event involved in the correlation shares a common field value. What we're looking for is a way to correlate based on a specified (likely very small) span of time between events of two or more sourcetypes.

Anyone have a good way of doing this? Thanks in advance,

-Frank

Re: Group Events Based Solely on Temporal Proximity

gkanapathy — Fri, 03 Sep 2010 01:06:16 GMT

As a matter of fact transaction does not require items to share any common fields. You can just specify maxspan (time from first event) and/or maxpause (max time between events), and all events piped to it will be grouped by those parameters.

I suppose I should note that transaction is mostlyuseful for finding out and grouping events in aggregate. If you're looking at a specific set of events in a specific time range, it is much more efficient specify your time range (or use a subsearch to generate a time range around a specific event) and use the search terms to get the sourcetypes you're interested in seeing.

Re: Group Events Based Solely on Temporal Proximity

fervin — Fri, 03 Sep 2010 02:31:27 GMT

Thanks for the prompt response. So, you're right about transaction not requiring common fields, but I can't see how to force it to give me what I need for this use case. It's very likely that I'm missing something incredibly simple...

An example:

sourcetype=ips OR sourcetype=syslog | transaction maxpause=1s

This could potentially give me bundles of all syslog, with no ips events.

What I want would be the functional equivalent of something like this fake query:

sourcetype=syslog | near sourcetype=ips maxpause=1s

Does this make sense?

Re: Group Events Based Solely on Temporal Proximity

hulahoop — Sat, 04 Sep 2010 09:29:05 GMT

How about the search command kmeans?

http://www.splunk.com/base/Documentation/latest/SearchReference/Kmeans

Re: Group Events Based Solely on Temporal Proximity

Stephen_Sorkin — Sat, 04 Sep 2010 10:53:31 GMT

I would use transaction as gkanapathy suggests in his answer. You can then post-filter those transactions to only include those with BOTH sourcetypes. For example:

sourcetype=ips OR sourcetype=*syslog* | transaction maxpause=1s | search sourcetype=ips sourcetype=*syslog*