Splunk Search

streamstats confusion

a_splunk_user
Path Finder

Having some trouble with streamstats.

I need to be alerted, once, at the time when a logical drive becomes less that 10% available. So, I have a script that writes a log file on a 5min interval, monitored by Splunk as sourcetype 'drivetracker'. I need the query to one-time alert me when the freePercent falls below the 10% mark. The source data looks correct.

Problem is, the Splunk query is not giving me the lastFreePercent figure based on the freePercent figure from the previous log file?

Here is my query:

sourcetype="drivetracker"  devId="*" 
| streamstats current=f window=1 last(freePercent) as lastFreePercent
| where freePercent<10 AND lastFreePercent>9
| table _time, server, devId, freePercent, lastFreePercent

and here are the results:

server  devId   totSpace    usedSpace   frSpace freePercent lastFreePercent
server1 C:  100 96.76   3.24    3   **98**
server1 E:  800 782.75  17.25   2   **76**
server1 F:  800 768.09  31.91   4   **24**

Apologies for the poor formatting. According to this, the three drives on server1 suddenly dropped a whole lot of space in under 5 minutes, which simply isn't the case.

Any help is appreciated!

Thanks!

Tags (1)
0 Karma

somesoni2
Revered Legend

Can you give this a try...

Updated answer:

sourcetype="drivetracker"  devId="*"  | sort -_time
| streamstats current=f window=1 last(freePercent) as lastFreePercent by devId,server
| where freePercent<10 AND lastFreePercent>9
| table _time, server, devId, freePercent, lastFreePercent

Give this a try as well

sourcetype="drivetracker" devId="*" | sort server,devId,-_time | eval devId_server = devId . server | streamstats current=f window=2 last(freePercent) as lastFreePercent by devId_server | where freePercent<10 AND lastFreePercent>9 | table _time, server, devId, devId_server, freePercent, lastFreePercent

a_splunk_user
Path Finder

The second query is pretty close, however the lastFreePercent shows the updated figure before the freePercent. Despite this being sorted by server or _time. Looking at http://answers.splunk.com/answers/105733/streamstats-is-reversed to see if that helps me. Thanks for all your help so far!

0 Karma

somesoni2
Revered Legend

Give the new answer a try...

0 Karma

a_splunk_user
Path Finder

By the way, this seems to be pretty close, when I aggregate the server & devId and streamstats by that, but only when I specify the a server & devId:

sourcetype="drivetracker" devId="*" server="server1" devId="C:" | sort -_time | eval devId_server = devId . server | streamstats current=f window=2 last(freePercent) as lastFreePercent by devId_server | where freePercent<10 AND lastFreePercent>9 | table _time, server, devId, devId_server, freePercent, lastFreePercent

Removing [server="server1" devId="C:"] causes a zero result set. Weird!

0 Karma

a_splunk_user
Path Finder

Interesting - when I ran the query you updated the lastFreePercent field was returned null values (returning zero records based on the filter). Removing the lastFreePercent<x from the filter yielded results, with the current freePercent figure as accurate, it seems.

The test showed that the source data appears to be correct.

0 Karma

somesoni2
Revered Legend

Also try the updated answer, just now seen that the grouping was missing from streamstats.

0 Karma

somesoni2
Revered Legend

Can you validate if the source data looks correct by executing this..

sourcetype="drivetracker" server="server1" devId="C:" OR devId="E:" OR devId="F:" | streamstats count by devId | where count ❤️

This should give you last 2 records for devId C:, E: and F: for server 1. Look at the value (if possible, provide it in the post) and see if really the 2nd last record says freepercent so high.

0 Karma

a_splunk_user
Path Finder

Thanks for the quick(!) response. Unfortunately the problem persists.

0 Karma
Get Updates on the Splunk Community!

Announcing Scheduled Export GA for Dashboard Studio

We're excited to announce the general availability of Scheduled Export for Dashboard Studio. Starting in ...

Extending Observability Content to Splunk Cloud

Watch Now!   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to leverage ...

More Control Over Your Monitoring Costs with Archived Metrics GA in US-AWS!

What if there was a way you could keep all the metrics data you need while saving on storage costs?This is now ...