Does Splunk REST API provide searches with cursor-like locators for the event stream?

[perichandra]

    We would like our application to pull events from a given customer's Splunk instance instead of forwarding. Forwarding was rejected due to problems with the administration of forwarder configurations from our side whereas the search querying leaves our side with complete control over any changes to the queries and does not require access to customer Splunk devices for update propagation. 

 So we plan to do this by periodically querying the Splunk instance with the REST API at short intervals to get large batches of events at a time. But we need to make sure we do not miss any events or get events in duplicate. So we need to have some cursor-like position specifiers to use each time we query. 

 Other devices like CISCO provide such capabilities which are already utilized well by us. So does Splunk also support such a model of interaction which is expected to be functionally equivalent to forwarding?

  Is it meaningful to talk of a unique location in an event stream in a Splunk instance which aggregates events from multiple device types and multiple instances of them? Any reader program from our app will be any time interested in only events from devices of a particular device type. But many such readers could be hitting the Splunk instance with pretty much the same query except for the device type. So one such reader for each device type.

If so, what is the programming model: what specific REST API calls and parameters are the ones to use for uniquely specifying the location in an event stream? 

What are the performance parameters? Also is the Splunk Java SDK the right one to use to make such REST API search calls from our application's event reader? Is this performance-wise a scalable model? 

Thanks in advance.

[psanford_splunk]

You could do it using the Java SDK, but you would have to implement the "cursor-like" functionality yourself.

You could also take a look at the shuttl app that was designed to help with scenarios like this. The app is here: http://splunk-base.splunk.com/apps/58003/shuttl

[psanford_splunk]

You will still need write your own logic to handle the processing of results you get in the specific time window to hit what I understand to be your use case. For example, "where was I in the processing of the data if there is an app crash or network interruption, etc. " You could definitely use the Java SDK to pull results and then write your own code to handle the cursor, position in the data stream logic.

[perichandra]

Hi Sanford,
Thanks for the response. It looks like the REST API provides the starttime /endtime pair and the earliest/latest pairs for this purpose.

From http://docs.splunk.com/Documentation/Splunk/5.0.3/SearchReference/Search#Examples :

Syntax: starttime= | endtime= | earliest= | latest=

Description: Specify start and end times using relative or absolute time.

So I can start asking for the next batch whose starttime is the time of the last event in the last batch of results.