Determining the total storage space a user's search artifacts are occupying (and calculating the time a search expires)

[BernardEAI]

I'm working on calculating the storage space taken up by a specific user. I would like to calculate the total size of their search artifacts at any given time - we would like to see if they are hitting their storage limits regularly. This will indicate we need to increase the storage limit.

I'm running this search to get the jobs for a specific user:

| rest /services/search/jobs | search author=<user> 

The next thing to do would be to calculate the size of all jobs that have not expired yet. I can get the life of the search (ttl field) and when the search started (published field), and a sum of those should give the time the search expires. However, if I check the "Jobs" report, the "Expires" field does not correspond to what I calculate. There must be some additional factor involved in the calculation of the "Expired" field..?

[BernardEAI · Accepted solution]

It turns out the Expired time is calculated by adding the ttl to start time of the search, plus the search duration. The search duration is the time the search takes to complete and display on screen. If there are many results and fields (as is the case with this search) the time to display on screen is significant. Therefor the Expired time is a bit more than:

search start + search duration + ttl

In any case, I discovered that I do not need to calculate size of the searches that have not expired yet, because only searches that have not expired yet are returned with the search:

| rest /services/search/jobs | search author=<user> 

 So I can just total the diskUsage field to get the total size of search artifacts for that user.

[isoutamo]

Time by time splunk returned also those rows which has ttl = 0 (which has expired). That is quite short period, but this can mix up your calculations.

[BernardEAI]

Thanks @isoutamo 

I have tested this a bit and I do not get any searches that has ttl = 0. Maybe I just haven't encountered that case yet in my testing.

[isoutamo]

Hi

If I looked those fields correctly, then ttl field directly told to you how much time the query results are available. So you could do something like

| rest /services/search/jobs
| fields author diskUsage ttl
| where ttl > 0
| stats count sum(diskUsage) as diskUsage by author

r. Ismo