Splunk Search

How to get an accurate timechart count of matching events if an extract is off by a second in a given polling interval?

det0n8r
Explorer

I'm struggling with counting session table exports that dump active sessions every five minutes. Basically I keep running into a problem where the count overlaps with a previous export every few intervals.

The data looks something like this:

Time="3:00:00.000 PM" User="user 1"
Time="3:00:00.000 PM" User="user 2"
Time="3:00:00.000 PM" User="user 3"
Time="3:05:00.000 PM" User="user 1"
Time="3:05:00.000 PM" User="user 2"
Time="3:10:00.000 PM" User="user 1"

Here's a sample search:

... | timechart span=5m count(_raw) as ActiveSessions

Which results in the attached visualization screenshot, where the data overlaps at certain points and incorrectly counts events from the previous polling interval.

I'm guessing that this is because the exports aren't running/completing exactly every five minutes, and so the span is intermittently counting two sets of exports.

For example, how do you do a proper count if an extract is off by a second in a given polling interval; like this:

Time="3:00:00.000 PM" User="user 1"
Time="3:00:00.000 PM" User="user 2"
Time="3:00:00.000 PM" User="user 3"
Time="3:05:01.000 PM" User="user 1"
Time="3:05:01.000 PM" User="user 2"
Time="3:10:00.000 PM" User="user 1"

Is there another way to get at this metric? I started looking into concurrency, but didn't have much luck.

0 Karma
1 Solution

masonmorales
Influencer

Normalize the times using "bin", dedup, then chart it.
i.e.
| bin _time span=5m | dedup _time user | timechart span=5m count as ActiveSessions

I'm not sure how you're getting the data into Splunk (DB Connect 2?), but the other option is to fix it on the ingest side. 😉

Hope this helps!

View solution in original post

masonmorales
Influencer

Normalize the times using "bin", dedup, then chart it.
i.e.
| bin _time span=5m | dedup _time user | timechart span=5m count as ActiveSessions

I'm not sure how you're getting the data into Splunk (DB Connect 2?), but the other option is to fix it on the ingest side. 😉

Hope this helps!

det0n8r
Explorer

Thank you sir! Using bin followed by a dedup definitely fixed the overlapping count!

To answer your question on the ingest method, these extracts are captured as standard output from a PowerShell script that is executed by the universal forwarder, and I suspect that performance issues on the SDK calls to the data source are to blame for the slight deviations in the interval duration.

0 Karma

masonmorales
Influencer

Would you mind posting that as a new question please?

0 Karma

det0n8r
Explorer
0 Karma
Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...