Splunk Enterprise

Reduce return value from API /search/jobs infromation

stonefr33
Explorer

Hi

Only just started using the API but have been unable to track down documentation on how to exclude fields from the job details return values.

Querying the /services/search/jobs with the header 'search=sid=<xxxxx.xxxxx>' which is working fine but is returning too much information under to content, in particular content/dict/phase0 and /remotesearch. Small searches are ok but when its a big search these can be >10 KB in size and takes several seconds to return. 

Tried defining fields in the header summerize=true, f=author but none of this seems to help reduce the return size. I'm only really after 6-8 fields so don't really need 90+ of the data that is returned

https://docs.splunk.com/Documentation/Splunk/8.0.5/RESTREF/RESTsearch#search.2Fjobs

Thanks

Labels (2)
Tags (1)
0 Karma

to4kawa
SplunkTrust
SplunkTrust

use table command

0 Karma

stonefr33
Explorer

It's not the search that I am looking to update its the job details you get back through the API, if you look at the job inspector you can see the components litsearch, phase0 and phase1. When running a subsearch these can become very large, basically querying the API to see if the job is running/done/paused etc. and who created it.

Working on submitting 200+ searches but it can take 5-6 seconds to get the job details back to know if it has completed. When you have 200 jobs that 5 seconds builds up

0 Karma

to4kawa
SplunkTrust
SplunkTrust
| rest /services/search/jobs
| fields - is* custom.* performance.* request.* runtime.* c* sample* eai.acl.perms.*
| table published id pid phase* dispatchState diskUsage search priority runDuration result* *

and report or alert(e.g. runDuration >60)

If you want to know the status of your search, how about this?

0 Karma

stonefr33
Explorer

That works when running the query from splunk to limit the data viewed, but the search itself takes several seconds. I could submit a new search and stream the results, will test how this performs vs. what I have currently

What I am aiming to do is to write a powershell module using the rest API, so I want to be able to run get-splunksearch this will return the sid, owner, expiry date, runtime and status. This is working but just seems very slow due to the amount of data that the jobs endpoint returns.

Code is very simple 

  $body = @{
                    search = ('sid={0}' -f $searchSid)
                }

Invoke-RestMethod -Uri https://servername:8089/services/search/jobs -Method GET -Credential $Credential -Body $body -erroraction stop

same as curl 
https://localhost:8089/services/search/jobs --get
-d sid=123454678.123456
 
I was hoping there would be a way to get the API to do the equivalent of 
| table published id pid phase* dispatchState diskUsage search priority runDuration result* *
0 Karma

stonefr33
Explorer

Just tested and its slightly slower to submit a search and stream the results instead of querying the API

$body = @{search = '|rest /services/search/jobs/1597138394.9683341 |fields sid, author, dispatchState, ttl, search, title, is*, resultcount | table sid, author, dispatchState, ttl, search, title, is*, resultcount'}
$url = 'https://{0}:8089/services/search/jobs/export'

Invoke-RestMethod -Uri $url -Method POST -Credential $stdcred -Body $body

1.2 Seconds

Vs.

$url=  'https://{0}:8089/services/search/jobs'
Invoke-RestMethod -Uri "$url/$searchsid" -Method GET -Credential $stdcred


Invoke-RestMethod -Uri "$searchURI/$searchsid" -Method GET -Credential $stdcred -erroraction

.7 seconds 

50% of the way through my searches so these times fluctuate based on the return from the subsearch <1second to >5seconds