Audit Log: Can someone confirm that cache_size value associated with a search_id is actually the percentage of free space relative to disk quota set in the user's role?

[Runals]

I'm trying to do some work related to watching the disk allocation associated for any particular user. Through some convoluted searches it appears that the cache_size value associated with a search_id is actually the percentage of free space relative to disk quota as set in the user's role checked before the search is actually run (I guess technically the lowest value across all the roles assigned to a user). Can anyone confirm that by chance?

To find the logs in question you can do the following search

index=_audit user=* action=quota cache_size=*

To test this I did a rest search on the jobs endpoint, converted the cumulative diskUsage to MB, and then subtracted that from the disk quota assigned to my test user. All searches were ad-hoc so had a 10 min TTL. At any rate the values seemed to be within reasonable range of each other.

[cpride_splunk · Accepted solution]

It looks like there is a log message in category "DispatchManager" at level "INFO" that will log the current usage and the quota when we check the quota before dispatching a job.

12-10-2014 17:45:32.362 -0800 INFO DispatchManager - Checking search quota: usage.concurSearches=1, quota.concurSearches=50, usage.realtimeSearches=0, quota.realtimeSearches=100, usage.diskUsageMB=11, quota.diskUsageMB=10000

You'll need to add this line to log.cfg:

category.DispatchManager=INFO

As far as what the log message is communicating, it is the disk space used for a given user as far as the quota enforcement in splunk is concerned (triggered when that user has the quota checked). It unfortunately doesn't have the user in it. However I think you can piece together the info you want assuming you have the INFO level messages with a search like this:

index=_internal sourcetype=splunkd component=DispatchManager log_level=INFO (enforceQuotas OR diskUsageMB) | rex field=message "enforceQuotas((?[^,]*)," | transaction maxevents=2 startswith="enforceQuotas" endswith="diskUsageMB" | timechart avg(usage_diskUsageMB) by user

This uses a second message that will always be output before we check the quota.

[Runals]

Is this a 6.2 thing - or at least post 6.1.3?; I'm not seeing it in my 6.1.3 system. I'm guessing/hoping the diskUsageMB is related back to the user vs simply the size of the available space being used. I'm also not seeing anything to link this back to a specific user which is what I'm hoping to see. There are certainly uses for looking at this at a system level but isn't what I'm looking for in terms of my use case. What I don't want to have to resort to doing is run a scheduled REST search every 10 minutes to get the diskUsage per user.

[cpride_splunk]

It looks like it has been around for quite a while, I had to add this line to log.cfg:

category.DispatchManager=INFO

As far as what the log message is communicating, it is the disk space used for a given user as far as the quota enforcement in splunk is concerned (triggered when that user has the quota checked). I agree it is unfortunate that this message doesn't have the user in it. However I think you can piece together the info you want assuming you have the INFO level messages with a search like this:

index=_internal sourcetype=splunkd
component=DispatchManager
log_level=INFO (enforceQuotas OR
diskUsageMB) | rex field=message
"enforceQuotas((?[^,]*)," |
transaction maxevents=2
startswith="enforceQuotas"
endswith="diskUsageMB" | timechart
avg(usage_diskUsageMB) by user

[Runals]

Ah - hadn't thought about the log.cfg. Will have to monkey around with this. Appreciate it!

[arahut_splunk]

cache_size is the raw # of searches that are cached in-memory in splunkd Quota Cache. It is for making quota checks faster.
It is not a percentage

[Runals]

I'll accept the answer; too bad that isn't related to solving my use case.