Getting Data In

Universal Forwarder Agent sends NO data on Windows disks shortly after they become full (100%)

amnonh
Explorer

We have correctly reporting Universal Forwarder agents running (Windows in this case) but whenever a local disk of the server that the agent is running on reached 100% occupancy (for a little while), there is no longer data coming in from the UF agent. If you look on the local server with Performance Monitor (LogicalDisk\% Free Space) and check the full disk in question, performance monitor shows 0.000. But in Splunk no data (not even that 0.000, see picture on the right side) is coming in anymore and our dashboard graphs that show disk occupancy turn blank as data stops flowing in (see picture on the left side). When you create space on the disk, even if it's still 99% filled, data starts flowing in again.

How can one work around this in Splunk, so when no data comes in where previously it was 99%, Splunk shows 100% instead of nothing at all...

This is the SPL in question (see bottom of picture for table output)

index="uf_basickpi" source="Perfmon:LogicalDisk" counter="% Free Space" instance!=HarddiskVolume* instance!=_Total host=SERVERNAME
| lookup resource_thresholds.csv resource_name as host, resource_metric as counter, resource_disk_instance as instance output resource_threshold_warning, resource_threshold_critical
| eval spaceFree=round(Value,0)
| eval spaceUsed=100-spaceFree
| timechart span=5m avg(spaceUsed) as "% Space Used", latest(resource_threshold_warning) as "Warning", latest(resource_threshold_critical) as "Critical" avg(spaceFree) as "% Space Free" by instance

 

Labels (2)
0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to July and August Tech Talks, Office Hours, and Webinars!

Dive into our sizzling summer lineup for July and August Community Office Hours and Tech Talks. Scroll down to ...

Edge Processor Scaling, Energy & Manufacturing Use Cases, and More New Articles on ...

Splunk Lantern is a Splunk customer success center that provides advice from Splunk experts on valuable data ...

Get More Out of Your Security Practice With a SIEM

Get More Out of Your Security Practice With a SIEMWednesday, July 31, 2024  |  11AM PT / 2PM ETREGISTER ...