Creating An Alert Based Off A Forwarded Text File

[DaveyJones]

Good Day All,

     I'm looking for assistance on how to create a Triggered Alert when a certain percentage number in a text .log file is met in real-time. For background, on a remote server there's a PowerShell script that runs locally via Task Scheduler set to daily which generates a text .log file containing the used percentage of that drive (F: Drive in this instance). The Data Inputs –> Forwarded Inputs –> Files & Directories on splunk along with the Universal Forwarder on that remote server are configured and the text .log file can be read in splunk when searched as shown below:

Search:

index=”main” source=”C:\\Admin\StorageLogs\storage_usage.log”

Result:

Event: 03/03/2025 13:10:40 - F: drive usage 17.989% used (All the text contained in the .log file)
source = C:\\Admin\StorageLogs\storage_usage.log   sourcetype = storage_usage-too_small 

     What would be the best way to go about setting up a triggered alert that notifies you in real-time when that text .log file meets/exceeds 75% of the F: drive used? I attempted saving it as an alert from there by performing the following:

Save As -> Alert:

Title: Storage Monitoring
Description: (Will add at the end)
Permissions: Shared in App
Alert Type: Real-time
Expires: 24 Hours
Trigger Conditions: Custom
Trigger alert when: (This is the field I’m trying to articulate the reading/notifying the 75% used part but unfamiliar with what to put)
In: 1 minute
Trigger: For each result
Throttle: (Unsure if needs to be enabled or not)
Trigger Actions -> When triggered -> Add to Triggered Alerts -> Severity: Medium

     Would it be easier to configure the reading/notifying when 75% used part in the trigger conditions above or by adding the inputs in the main search query then saving? My apologies if I’m incorrect in any of my interpretations/explanations, I just started with this team and have basically no experience with splunk. Any information or guidance is greatly appreciated, thanks again.

[richgalloway · Accepted solution]

Use the rex command to extract the usage value then test the value to see if the alert should be triggered.  I find it more reliable to put the threshold in the alert rather than in the metadata.

index=”main” source=”C:\\Admin\StorageLogs\storage_usage.log”
| rex "usage (?<usage>[^%]+)% used"
| where usage > 75

[DaveyJones]

Good day, unfortunately this did not prompt a triggered alert even after changing the usage value to a lower number when testing it. Thank you though.

[richgalloway]

I left out a character.  Try my updated query.

[DaveyJones]

Thank you for the edit, I got it to work after adding a : after usage as without it nothing was generating. Thank you for your assistance

index=”main” source=”C:\\Admin\StorageLogs\storage_usage.log” | rex "usage: (?<usage>[^%]+)% used" | where usage >= 75

[livehybrid]

Hi @DaveyJones 

I think the easiest way to achieve this might be to add the following to your search

| rex field=_raw "usage (?<diskUsage>[0-9\.]+)% used"
| where diskUsage>75

Adjust the diskUsage>75 to whatever you need. This works by extracting the % value of the disk usage from the raw event and then only returning events where the diskUsage is over the specified value.

You would then create the alert to run on a cron-schedule as required, such as every hour (Real-Time is generally not advised, especially as disk usage shouldnt drastically change that quick! So maybe run on a suitable interval, and adjust the time it looks back over (earliest) accordingly.

Set the alert to Trigger alert when: Number of Results, is greater than 0.

This will then trigger the alert if there is any result from the search (which has the specified limit on it).

Please let me know how you get on and consider adding karma to this or any other answer if it has helped.
Regards

Will

[DaveyJones]

Good day, unfortunately this did not prompt a triggered alert even after changing the usage value to a lower number to test it. Thank you though.

[livehybrid]

Hi @DaveyJones 

Please could you provide the search you ended up using so I can look into this further for you?

Thanks

Will

[DaveyJones]

Good day,

I got it to work after adding a : after "usage" as shown below:

index=”main” source=”C:\\Admin\StorageLogs\storage_usage.log” | rex field=_raw "usage: (?<diskUsage>[0-9\.]+)% used" | where diskUsage>75

Thank you for your assistance.