Splunk Search

Determining the total storage space a user's search artifacts are occupying (and calculating the time a search expires)

BernardEAI
Communicator

I'm working on calculating the storage space taken up by a specific user. I would like to calculate the total size of their search artifacts at any given time - we would like to see if they are hitting their storage limits regularly. This will indicate we need to increase the storage limit.

I'm running this search to get the jobs for a specific user:

| rest /services/search/jobs | search author=<user> 

The next thing to do would be to calculate the size of all jobs that have not expired yet. I can get the life of the search (ttl field) and when the search started (published field), and a sum of those should give the time the search expires. However, if I check the "Jobs" report, the "Expires" field does not correspond to what I calculate. There must be some additional factor involved in the calculation of the "Expired" field..?

Labels (1)
0 Karma
1 Solution

BernardEAI
Communicator

It turns out the Expired time is calculated by adding the ttl to start time of the search, plus the search duration. The search duration is the time the search takes to complete and display on screen. If there are many results and fields (as is the case with this search) the time to display on screen is significant. Therefor the Expired time is a bit more than:

search start + search duration + ttl

In any case, I discovered that I do not need to calculate size of the searches that have not expired yet, because only searches that have not expired yet are returned with the search:

| rest /services/search/jobs | search author=<user> 

 So I can just total the diskUsage field to get the total size of search artifacts for that user.

View solution in original post

0 Karma

BernardEAI
Communicator

It turns out the Expired time is calculated by adding the ttl to start time of the search, plus the search duration. The search duration is the time the search takes to complete and display on screen. If there are many results and fields (as is the case with this search) the time to display on screen is significant. Therefor the Expired time is a bit more than:

search start + search duration + ttl

In any case, I discovered that I do not need to calculate size of the searches that have not expired yet, because only searches that have not expired yet are returned with the search:

| rest /services/search/jobs | search author=<user> 

 So I can just total the diskUsage field to get the total size of search artifacts for that user.

0 Karma

isoutamo
SplunkTrust
SplunkTrust
Time by time splunk returned also those rows which has ttl = 0 (which has expired). That is quite short period, but this can mix up your calculations.
0 Karma

BernardEAI
Communicator

Thanks @isoutamo 

I have tested this a bit and I do not get any searches that has ttl = 0. Maybe I just haven't encountered that case yet in my testing.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Hi

If I looked those fields correctly, then ttl field directly told to you how much time the query results are available. So you could do something like

 

| rest /services/search/jobs
| fields author diskUsage ttl
| where ttl > 0
| stats count sum(diskUsage) as diskUsage by author

 

r. Ismo 

0 Karma
Get Updates on the Splunk Community!

Enterprise Security Content Update (ESCU) v3.54.0

The Splunk Threat Research Team (STRT) recently released Enterprise Security Content Update (ESCU) v3.54.0 and ...

Using Machine Learning for Hunting Security Threats

WATCH NOW Seeing the exponential hike in global cyber threat spectrum, organizations are now striving more for ...

New Learning Videos on Topics Most Requested by You! Plus This Month’s New Splunk ...

Splunk Lantern is a customer success center that provides advice from Splunk experts on valuable data ...