We have correctly reporting Universal Forwarder agents running (Windows in this case) but whenever a local disk of the server that the agent is running on reached 100% occupancy (for a little while), there is no longer data coming in from the UF agent. If you look on the local server with Performance Monitor (LogicalDisk\% Free Space) and check the full disk in question, performance monitor shows 0.000. But in Splunk no data (not even that 0.000, see picture on the right side) is coming in anymore and our dashboard graphs that show disk occupancy turn blank as data stops flowing in (see picture on the left side). When you create space on the disk, even if it's still 99% filled, data starts flowing in again.
How can one work around this in Splunk, so when no data comes in where previously it was 99%, Splunk shows 100% instead of nothing at all...
This is the SPL in question (see bottom of picture for table output)
index="uf_basickpi" source="Perfmon:LogicalDisk" counter="% Free Space" instance!=HarddiskVolume* instance!=_Total host=SERVERNAME
| lookup resource_thresholds.csv resource_name as host, resource_metric as counter, resource_disk_instance as instance output resource_threshold_warning, resource_threshold_critical
| eval spaceFree=round(Value,0)
| eval spaceUsed=100-spaceFree
| timechart span=5m avg(spaceUsed) as "% Space Used", latest(resource_threshold_warning) as "Warning", latest(resource_threshold_critical) as "Critical" avg(spaceFree) as "% Space Free" by instance
