topic Re: Event correlation question in Splunk Search

Event correlation question

pjmenon — Tue, 15 Jun 2010 22:28:00 GMT

I am evaluating SPLUNK for my client. Reading previous questions tells me I can do this, but want to confirm.

have 2 dissimilar log files

file1 : field1, 2, 3, time1...
file2 : field1, a, b, time2....

If pattern1 in file 1, find pattern2 in file2 where field1 matches and time1 , time 2 are within n seconds of each other.

Re: Event correlation question

Lowell — Wed, 16 Jun 2010 00:09:43 GMT

I'm not sure about the pattern1 and pattern2 thing exactly--if you gave an example someone could respond with more confidence, but most likely it's possible to match these using a simple search terms, field matching or if necessary, a regular expression match.

The rest of it is certainly possible.

You would probably end up with a search that looks something like this:

(source=file1 pattern1) OR (source=2 pattern2) | transaction fields="field1" maxspan=10s

If you then want to make sure that you have both sources in your resulting transaction, you could simply add an additional "search" command after you build your transactions.

(source=file1 pattern1) OR (source=2 pattern2) | transaction fields="field1" maxspan=10s | search source=file1 source=file2

If pattern1 and pattern2, for whatever reason, must be regular expressions, then you could use a search like this: (Note: This search will not be as fast as the first approach and is generally more difficult to maintain (since not everyone is a regex-master). It's generally recommended to use normal search terms rather than using a regex match, but it can be done.)

source=file1 OR source=file2 | regex _raw="^(pattern1|pattern2)$" | transaction fields="field1" maxspan=10s

I hope this gives you an idea of the flexibility that's available. You can combine tons of commands to get splunk to do almost anything you want it to do. And if you can't find an existing search command, you can actually write your own too. -- I don't recommended starting there, but it's nice to know you have options.

Re: Event correlation question

pjmenon — Wed, 16 Jun 2010 08:01:02 GMT

I think you may have answered my question. Patetrn1 & 2 can be simple regexes. I think the above search matches 2 events even from the same file (say file1) if they are within 10s. How to search a pair of events each of them coming from a different file?
Thank you !

Re: Event correlation question

Lowell — Wed, 16 Jun 2010 22:16:14 GMT

If pattern 1 and 2 are simple regex than additional steps would be required. (Normally you can come up with a search expression that is simpler and more effective than a regex, but you can use a regex if you have too.)