Splunk Search

Correlation known events with all other sources

Cuyose
Builder

So I have a search that can be run that returns 12 events over a 60 minute period from a single known source. I would like to run a general query that correlates all other events from all sources that also occur 12 times in the 60 minute period.

The hypothesis being that the error I am searching for may also have events somewhere that correlate, however it's difficult to go through dozens of source-types and events manually doing this.

Tags (1)
0 Karma

aholzer
Motivator

Just widen your base search.

If originally you were doing something like:

index="myindex" sourcetype="mysourcetype"

Try:

index="myindex" | transaction maxspan=1m maxpause=5s

This will take all events from your index and make transactions of 1 minute length with a maximum time between events of 5 seconds.

If your search needs to cross multiple indexes, just list all the indexes with OR statements out before the pipe:

index="myindex1" OR index="myindex2" OR...

0 Karma

kristian_kolb
Ultra Champion

You could have a look at the sparkline feature for stats or chart.

 earliest = -1h | stats sparkline c  by sourcetype

which could give you a quick overview of things that seem to be happening at the same time.

http://docs.splunk.com/Documentation/Splunk/5.0.5/Search/Addsparklinestosearchresults

/K

0 Karma

Cuyose
Builder

All of the examples are assuming transform is being used on a single source, which I can do easily, however if I want to try to correlate on other events without common fields just by time, im drawing a blank

0 Karma

aholzer
Motivator

Look at the maxspan, maxwait, and similar attribute of the transaction command. One of those is probably what you are looking for

0 Karma

Cuyose
Builder

Yes, im trying to look into transaction, but there is no common field to be aggregated other than a +- 1 second interval, but im not sure how to employ that.

For example I have an error log that 5% of the time returns a stack trace. I generate a search to isolate those events. I want to be able to run something that figures out what other events across all sources in the same index correlate most closely to those "5%" events based on the timestamp.

0 Karma

aholzer
Motivator

How about using stats by sourcetype or eventtype?

You can run two separate searches:

<base search...> | stats count by sourcetype | where count = 12

OR

<base search...> | stats count by eventtype | where count = 12

This won't give you the correlation you are looking for, but it will give you only sourcetypes or eventtypes that had 12 events in your time frame.

For correlation you may want to look into transactions, after you have identified a field that you can use to correlate all the events together.

0 Karma
Get Updates on the Splunk Community!

.conf24 | Registration Open!

Hello, hello! I come bearing good news: Registration for .conf24 is now open!   conf is Splunk’s rad annual ...

Splunk is officially part of Cisco

Revolutionizing how our customers build resilience across their entire digital footprint.   Splunk ...

Splunk APM & RUM | Planned Maintenance March 26 - March 28, 2024

There will be planned maintenance for Splunk APM and RUM between March 26, 2024 and March 28, 2024 as ...