Streaming realtime results via the REST API?

[nnachefski]

How do you stream real-time results via the rest api? I've tried using the typical search submit method, which always results in XML defining my search ID (sid).
I cannot poll the results with the typical method (/services/search/jobs/%s/results). How do you connect up and end-point so that i can start the stream?

P.S. I'm using pyCurl and SAX, which both support streaming results. I just cant figure out how to tell splunk to stream the results to me. Any pointers?

[steveyz]

You can also use the services/search/jobs/export endpoint. It can emit streaming results in xml or csv format (use 'output_mode' arg to specify)

[bsonposh]

For Realtime searches you need to pass search_mode. You can collect the events as you go or you can collect them all at the end.

This is still a work in progress but it illustrates what you after. In Powershell I am working on keeping track of what messages that have already been returned and what is new.

There is also a control feature that will allow you pause,unpause,finalize,cancel, and touch the search. I am investigating how to use that.

Here is example of how I did it in Powershell

function New-SplunkSearchJob
{

    [Cmdletbinding()]
    Param(

        [Parameter()]
        [String]$Server = $Splunk_Server,

        [Parameter()]
        [int]$Port = $Splunk_Port,

        # Search parameters support in POST
        # http://www.splunk.com/base/Documentation/latest/Developer/RESTSearch#POST

        [Parameter()]           # search
        [String]$Search = "search *",

        [Parameter()]           # required_field_list (comma separated list)
        [String]$RequireFields,

        [Parameter()]           # earliest_time
        [String]$StartDate,

        [Parameter()]           # latest_time
        [String]$EndDate,

        [Parameter()]           # id
        [String]$ID,

        [Parameter()]           # max_count = int
        [int]$MaxCount,

        [Parameter()]           # max_time = int
        [int]$MaxTime,

        [Parameter()]
        [System.Management.Automation.PSCredential]$Creds = $Splunk_Credentials,

        # Plan to implement

        #[Parameter()]           # exec_mode = blocking | oneshot | normal (only supporting oneshot/normal)
        #[Switch]$Wait,      

        [Parameter()]           # 'search_mode: normal | realtime'
        [Switch]$Realtime

    )

    function Get-Bytes($String)
    {
        [Byte[]]$byteArray = [System.Text.Encoding]::UTF8.GetBytes($String)
        $byteArray
    }

    $URL = "https://${Server}:${Port}/services/search/jobs" 

    Write-Verbose " [New-SplunkSearchJob] :: URL = $URL"

    $Request = [System.Net.WebRequest]::Create($url)
    $Request.Credentials = $Creds
    $Request.Method ="POST"
    $Request.ContentType = "application/x-www-form-urlencoded"
    $RequestStream = $Request.GetRequestStream()

    Write-Verbose " [New-SplunkSearchJob] :: Creating POST message"

    Write-Verbose " [New-SplunkSearchJob] :: Adding Search string [search=$Search] to POST message"
    #[string]$PostString = "search=$Search"
    [string]$PostString = "search={0}" -f [System.Web.HttpUtility]::UrlEncode($search)

    if($StartDate)
    {
        $PostString += "&earliest_time={0}" -f [System.Web.HttpUtility]::UrlEncode($StartDate)
    }

    if($EndDate)
    {
        $PostString += "&latest_time={0}" -f [System.Web.HttpUtility]::UrlEncode($EndDate)
    }

    if($MaxCount)
    {
        $PostString += "&max_count={0}" -f [System.Web.HttpUtility]::UrlEncode($MaxCount)
    }

    if($MaxTime)
    {
        $PostString += "&max_time={0}" -f [System.Web.HttpUtility]::UrlEncode($MaxTime)
    }
    if($Realtime)
    {
        $PostString += "&search_mode=realtime"
    }

    Write-Verbose " [New-SplunkSearchJob] :: `$PostString = $PostString"

    Write-Verbose " [New-SplunkSearchJob] :: Converting POST message to Byte Array"
    $Bytes = Get-Bytes $PostString

    Write-Verbose " [New-SplunkSearchJob] :: Sending POST message"
    $RequestStream.Write($Bytes,0,$Bytes.length)

    Write-Verbose " [New-SplunkSearchJob] :: Closing POST stream"
    $RequestStream.Close()

    Write-Verbose " [New-SplunkSearchJob] :: Getting Response from POST"
    $Response = $Request.GetResponse()
    $Reader = new-object System.IO.StreamReader($Response.GetResponseStream())

    [XML]$Results = $Reader.ReadToEnd()

    $SID = $Results.Response.sid
    Write-Verbose " [New-SplunkSearchJob] :: ID = $SID"

    if($SID)
    {
        if($Realtime)
        {
            while($true)
            {
                Get-SplunkSearchJobResult -SID $SID
                sleep 3
            }
        }
        else
        {
            $cont = $true
            Write-Host "Please wait. It could take a bit..." -NoNewline
            while($cont)
            {
                $Job = Get-SplunkSearchJob -Filter $SID | ?{$_.isDone -eq 1}
                if($Job.ID)
                {
                    $Job | Get-SplunkSearchJobResult 
                    $cont = $false
                    Write-Host
                    Write-Host "Search complete"
                    continue
                }
                sleep 1
            }
        }
    }
}    # New-SplunkSearchJob

[iinet]

Any idea where we would find the 'Get-SplunkSearchJob' command?

[sideview]

I dont think this gets used or talked about a lot, so mileage may definitely vary.

but note that in the search API, you can supply an 'exec_mode' argument. And if you set it to 'oneshot', it will return the data from the initial POST.

look in this page for exec_mode and oneshot: http://www.splunk.com/base/Documentation/latest/Developer/RESTSearch