Getting Data In

Streaming realtime results via the REST API?

nnachefski
Engager

How do you stream real-time results via the rest api? I've tried using the typical search submit method, which always results in XML defining my search ID (sid).
I cannot poll the results with the typical method (/services/search/jobs/%s/results). How do you connect up and end-point so that i can start the stream?

P.S. I'm using pyCurl and SAX, which both support streaming results. I just cant figure out how to tell splunk to stream the results to me. Any pointers?

Tags (3)

steveyz
Splunk Employee
Splunk Employee

You can also use the services/search/jobs/export endpoint. It can emit streaming results in xml or csv format (use 'output_mode' arg to specify)

bsonposh
Communicator

For Realtime searches you need to pass search_mode. You can collect the events as you go or you can collect them all at the end.

This is still a work in progress but it illustrates what you after. In Powershell I am working on keeping track of what messages that have already been returned and what is new.

There is also a control feature that will allow you pause,unpause,finalize,cancel, and touch the search. I am investigating how to use that.

Here is example of how I did it in Powershell

function New-SplunkSearchJob
{

    [Cmdletbinding()]
    Param(

        [Parameter()]
        [String]$Server = $Splunk_Server,

        [Parameter()]
        [int]$Port = $Splunk_Port,

        # Search parameters support in POST
        # http://www.splunk.com/base/Documentation/latest/Developer/RESTSearch#POST

        [Parameter()]           # search
        [String]$Search = "search *",

        [Parameter()]           # required_field_list (comma separated list)
        [String]$RequireFields,

        [Parameter()]           # earliest_time
        [String]$StartDate,

        [Parameter()]           # latest_time
        [String]$EndDate,

        [Parameter()]           # id
        [String]$ID,

        [Parameter()]           # max_count = int
        [int]$MaxCount,

        [Parameter()]           # max_time = int
        [int]$MaxTime,

        [Parameter()]
        [System.Management.Automation.PSCredential]$Creds = $Splunk_Credentials,

        # Plan to implement

        #[Parameter()]           # exec_mode = blocking | oneshot | normal (only supporting oneshot/normal)
        #[Switch]$Wait,      

        [Parameter()]           # 'search_mode: normal | realtime'
        [Switch]$Realtime

    )

    function Get-Bytes($String)
    {
        [Byte[]]$byteArray = [System.Text.Encoding]::UTF8.GetBytes($String)
        $byteArray
    }

    $URL = "https://${Server}:${Port}/services/search/jobs" 

    Write-Verbose " [New-SplunkSearchJob] :: URL = $URL"

    $Request = [System.Net.WebRequest]::Create($url)
    $Request.Credentials = $Creds
    $Request.Method ="POST"
    $Request.ContentType = "application/x-www-form-urlencoded"
    $RequestStream = $Request.GetRequestStream()

    Write-Verbose " [New-SplunkSearchJob] :: Creating POST message"

    Write-Verbose " [New-SplunkSearchJob] :: Adding Search string [search=$Search] to POST message"
    #[string]$PostString = "search=$Search"
    [string]$PostString = "search={0}" -f [System.Web.HttpUtility]::UrlEncode($search)

    if($StartDate)
    {
        $PostString += "&earliest_time={0}" -f [System.Web.HttpUtility]::UrlEncode($StartDate)
    }

    if($EndDate)
    {
        $PostString += "&latest_time={0}" -f [System.Web.HttpUtility]::UrlEncode($EndDate)
    }

    if($MaxCount)
    {
        $PostString += "&max_count={0}" -f [System.Web.HttpUtility]::UrlEncode($MaxCount)
    }

    if($MaxTime)
    {
        $PostString += "&max_time={0}" -f [System.Web.HttpUtility]::UrlEncode($MaxTime)
    }
    if($Realtime)
    {
        $PostString += "&search_mode=realtime"
    }

    Write-Verbose " [New-SplunkSearchJob] :: `$PostString = $PostString"

    Write-Verbose " [New-SplunkSearchJob] :: Converting POST message to Byte Array"
    $Bytes = Get-Bytes $PostString

    Write-Verbose " [New-SplunkSearchJob] :: Sending POST message"
    $RequestStream.Write($Bytes,0,$Bytes.length)

    Write-Verbose " [New-SplunkSearchJob] :: Closing POST stream"
    $RequestStream.Close()

    Write-Verbose " [New-SplunkSearchJob] :: Getting Response from POST"
    $Response = $Request.GetResponse()
    $Reader = new-object System.IO.StreamReader($Response.GetResponseStream())

    [XML]$Results = $Reader.ReadToEnd()

    $SID = $Results.Response.sid
    Write-Verbose " [New-SplunkSearchJob] :: ID = $SID"

    if($SID)
    {
        if($Realtime)
        {
            while($true)
            {
                Get-SplunkSearchJobResult -SID $SID
                sleep 3
            }
        }
        else
        {
            $cont = $true
            Write-Host "Please wait. It could take a bit..." -NoNewline
            while($cont)
            {
                $Job = Get-SplunkSearchJob -Filter $SID | ?{$_.isDone -eq 1}
                if($Job.ID)
                {
                    $Job | Get-SplunkSearchJobResult 
                    $cont = $false
                    Write-Host
                    Write-Host "Search complete"
                    continue
                }
                sleep 1
            }
        }
    }
}    # New-SplunkSearchJob

iinet
Engager

Any idea where we would find the 'Get-SplunkSearchJob' command?

0 Karma

sideview
SplunkTrust
SplunkTrust

I dont think this gets used or talked about a lot, so mileage may definitely vary.

but note that in the search API, you can supply an 'exec_mode' argument. And if you set it to 'oneshot', it will return the data from the initial POST.

look in this page for exec_mode and oneshot: http://www.splunk.com/base/Documentation/latest/Developer/RESTSearch

Get Updates on the Splunk Community!

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...

Let’s Get You Certified – Vegas-Style at .conf24

Are you ready to level up your Splunk game? Then, let’s get you certified live at .conf24 – our annual user ...