How to use the REST API to just run a search and stream the results back?

[a212830]

Hi,

I have a question about using the REST API to run a search. The doc seems to indicate that you need to follow 3 steps - create a search job, get the search status, and then get the search results. Is there any way to just run the search and stream the results back? Seems like a lot of steps...

[venugoski]

Need help in the splunk api curl query, i am seeing the below error. 

curl -k -u apiuser:password "https://10.236.141.0:8089/services/search/jobs/export" -d search="search index=address-validation earliest=-15m latest=now source=\"eventhub://sams-jupiter-prod-scus-logs-premium-1.servicebus.windows.net/list-service;\" | stats dc(kubernetes.pod_name) as pod_count"

<?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="ERROR">Unbalanced quotes.</msg>
</messages>
</response>

sometimes i dont see the result either:

curl -k -u user:password https://10.236.141.0:8089/services/search/jobs/export -d search="search index=address-validation earliest=-15m latest=now source=eventhub://sams-jupiter-prod-wus-logs-premium-1.servicebus.windows.net/address-validation; | stats dc(kubernetes.pod_name) as pod_count"



"<?xml version='1.0' encoding='UTF-8'?>

<results preview='0'>

<meta>

<fieldOrder />

</meta>

<messages>

  <msg type="INFO">Your timerange was substituted based on your search string</msg>

</messages>



</results>"

[venugoski]

i am running the below query 

curl -k -u user:password https://10.236.142.0:8089/services/search/jobs/export -d search="search index=list-service source="eventhub://sams-jupiter-prod-scus-logs-premium-1.servicebus.windows.net/list-service;" "kubernetes.namespace_name"="list-service" | stats dc(kubernetes.pod_name) as pod_count"

<?xml version='1.0' encoding='UTF-8'?>
<results preview='0'>
<meta>
<fieldOrder />
</meta>
</results>
zsh: command not found: kubernetes.namespace_name=list-service | stats dc(kubernetes.pod_name) as pod_count

[jwalthour]

In your search, you need to escape your quotes, like this:

 search="search index=list-service source=\"eventhub://sams-jupiter-prod-scus-logs-premium-1.servicebus.windows.net/list-service;\" \"kubernetes.namespace_name\"=\"list-service\" | stats dc(kubernetes.pod_name) as pod_count"

or use single quotes around the search contents:

search=‘search index=list-service source="eventhub://sams-jupiter-prod-scus-logs-premium-1.servicebus.windows.net/list-service;" "kubernetes.namespace_name”="list-service" | stats dc(kubernetes.pod_name) as pod_count’

[venugoski]

Need help in the splunk api curl query, i am seeing the below error. 

curl -k -u apiuser:password "https://10.236.141.0:8089/services/search/jobs/export" -d search="search index=address-validation earliest=-15m latest=now source=\"eventhub://sams-jupiter-prod-scus-logs-premium-1.servicebus.windows.net/list-service;\" | stats dc(kubernetes.pod_name) as pod_count"

<?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="ERROR">Unbalanced quotes.</msg>
</messages>
</response>

sometimes i dont see the result either:

curl -k -u user:password https://10.236.141.0:8089/services/search/jobs/export -d search="search index=address-validation earliest=-15m latest=now source=eventhub://sams-jupiter-prod-wus-logs-premium-1.servicebus.windows.net/address-validation; | stats dc(kubernetes.pod_name) as pod_count"

<?xml version='1.0' encoding='UTF-8'?>

<results preview='0'>

<meta>

<fieldOrder />

</meta>

<messages>

  <msg type="INFO">Your timerange was substituted based on your search string</msg>

</messages>

</results>

[arunslal]

Looks like spaces and quotes are being identified as shell.  Try escaping them like below:

curl -k -u user:password https://10.236.142.0:8089/services/search/jobs/export -d search='search index=list-service source=\"eventhub://sams-jupiter-prod-scus-logs-premium-1.servicebus.windows.net/list-service;\" \"kubernetes.namespace_name\"=\"list-service\" | stats dc(kubernetes.pod_name) as pod_count'

I had a very long query that needed to be passed via rest api. I ran into such issues but url encoding the query was very helpful.

I used this website for that:

https://meyerweb.com/eric/tools/dencoder/

[andrewlamonica]

I wasn't able to get the above example to work. But, this one worked fine for me...

curl -k -u 'username:password' https://splunk.host.name.here:8089/services/search/jobs/export  -d search="search error | head 3" -d output_mode=xml

The main differences are...
1. I needed to quote my username and password (as they have special chars in them)
2. I needed to replace "servicesNS" with just "services"
3. Having "search" in the URL twice didn't work for me, I removed on of them.
4. Only one output_mode can be specified at a time (I put "xml" in my example, but the other two work, just not all at once)
5. I needed to remove the "smart quotes" and use normal quotes. That might just be my console being picky, though.

[kutzi]

I needed to add -d exec_mode=oneshot  otherwise it wouldn't stream the results back.

E.g.

curl -k -u 'username:password' https://splunk.host.name.here:8089/services/search/jobs/export  -d search="search index=_internal | head 3" -d output_mode=csv -d exec_mode=oneshot

[adminpulse]

where should i run this command.

Also, please elaborate the command

curl -k -u 'username:password' https://splunk.host.name.here:8089/services/search/jobs/export  -d search="search index=_internal | head 3" -d output_mode=csv -d exec_mode=oneshot

[spunk311z]

@kutzi  thank you SO MUCH for posting this!!   

I have spent several hours trying to figure out how to do a basic synchronous search via curl/api (have tried 100s of curl command variations).

I have scripts working with the Async method (as that is clearly documented in splunk docs),  however im not sure why the direct, synchronous method seems to have little/no documentation.  (i realize the pros/cons of each and that synchronous search should rarely be used).  

Again thanks for taking the time to make this post,  it was super helpful.

Here is what is working for me:

curl -u admin:mypw -k https://splunk.me:8089/services/search/jobs/export -d search="search index=routers Web Down | head 3" -d output_mode=csv -d exec_mode=oneshot

### also this works:
curl -u admin:mypw -k https://splunk.me:8089/services/search/jobs/export -d output_mode=json -d search="search index=routers |head 10"

[neelamssantosh · Accepted solution]

curl -ku username:password https://SearchHead_host:8089/servicesNS/admin/search/search/jobs/export -d search=“search index%3D_internal | head 3” -d output_mode=csv/xml/json

[arunslal]

Hi @neelamssantosh  Apologies for asking as a comment. 

In case of multiserach, does the rest api call syntax differ? I'm able to do normal search with the above syntax but it fails when put a multisearch query instead. Can you please take a look? thank you.

[a212830]

Thanks! Works great.

[neeldesai1992]

the restful url call /serviceNS/admin/search/search/jobs/export is right?

[neeldesai1992]

Does this require to have saved search query? Or you are making it on demand?

[nopslide]

It's on demand