Getting Data In

Streaming realtime results via the REST API?

nnachefski
Engager

How do you stream real-time results via the rest api? I've tried using the typical search submit method, which always results in XML defining my search ID (sid).
I cannot poll the results with the typical method (/services/search/jobs/%s/results). How do you connect up and end-point so that i can start the stream?

P.S. I'm using pyCurl and SAX, which both support streaming results. I just cant figure out how to tell splunk to stream the results to me. Any pointers?

Tags (3)

steveyz
Splunk Employee
Splunk Employee

You can also use the services/search/jobs/export endpoint. It can emit streaming results in xml or csv format (use 'output_mode' arg to specify)

bsonposh
Communicator

For Realtime searches you need to pass search_mode. You can collect the events as you go or you can collect them all at the end.

This is still a work in progress but it illustrates what you after. In Powershell I am working on keeping track of what messages that have already been returned and what is new.

There is also a control feature that will allow you pause,unpause,finalize,cancel, and touch the search. I am investigating how to use that.

Here is example of how I did it in Powershell

function New-SplunkSearchJob
{

    [Cmdletbinding()]
    Param(

        [Parameter()]
        [String]$Server = $Splunk_Server,

        [Parameter()]
        [int]$Port = $Splunk_Port,

        # Search parameters support in POST
        # http://www.splunk.com/base/Documentation/latest/Developer/RESTSearch#POST

        [Parameter()]           # search
        [String]$Search = "search *",

        [Parameter()]           # required_field_list (comma separated list)
        [String]$RequireFields,

        [Parameter()]           # earliest_time
        [String]$StartDate,

        [Parameter()]           # latest_time
        [String]$EndDate,

        [Parameter()]           # id
        [String]$ID,

        [Parameter()]           # max_count = int
        [int]$MaxCount,

        [Parameter()]           # max_time = int
        [int]$MaxTime,

        [Parameter()]
        [System.Management.Automation.PSCredential]$Creds = $Splunk_Credentials,

        # Plan to implement

        #[Parameter()]           # exec_mode = blocking | oneshot | normal (only supporting oneshot/normal)
        #[Switch]$Wait,      

        [Parameter()]           # 'search_mode: normal | realtime'
        [Switch]$Realtime

    )

    function Get-Bytes($String)
    {
        [Byte[]]$byteArray = [System.Text.Encoding]::UTF8.GetBytes($String)
        $byteArray
    }

    $URL = "https://${Server}:${Port}/services/search/jobs" 

    Write-Verbose " [New-SplunkSearchJob] :: URL = $URL"

    $Request = [System.Net.WebRequest]::Create($url)
    $Request.Credentials = $Creds
    $Request.Method ="POST"
    $Request.ContentType = "application/x-www-form-urlencoded"
    $RequestStream = $Request.GetRequestStream()

    Write-Verbose " [New-SplunkSearchJob] :: Creating POST message"

    Write-Verbose " [New-SplunkSearchJob] :: Adding Search string [search=$Search] to POST message"
    #[string]$PostString = "search=$Search"
    [string]$PostString = "search={0}" -f [System.Web.HttpUtility]::UrlEncode($search)

    if($StartDate)
    {
        $PostString += "&earliest_time={0}" -f [System.Web.HttpUtility]::UrlEncode($StartDate)
    }

    if($EndDate)
    {
        $PostString += "&latest_time={0}" -f [System.Web.HttpUtility]::UrlEncode($EndDate)
    }

    if($MaxCount)
    {
        $PostString += "&max_count={0}" -f [System.Web.HttpUtility]::UrlEncode($MaxCount)
    }

    if($MaxTime)
    {
        $PostString += "&max_time={0}" -f [System.Web.HttpUtility]::UrlEncode($MaxTime)
    }
    if($Realtime)
    {
        $PostString += "&search_mode=realtime"
    }

    Write-Verbose " [New-SplunkSearchJob] :: `$PostString = $PostString"

    Write-Verbose " [New-SplunkSearchJob] :: Converting POST message to Byte Array"
    $Bytes = Get-Bytes $PostString

    Write-Verbose " [New-SplunkSearchJob] :: Sending POST message"
    $RequestStream.Write($Bytes,0,$Bytes.length)

    Write-Verbose " [New-SplunkSearchJob] :: Closing POST stream"
    $RequestStream.Close()

    Write-Verbose " [New-SplunkSearchJob] :: Getting Response from POST"
    $Response = $Request.GetResponse()
    $Reader = new-object System.IO.StreamReader($Response.GetResponseStream())

    [XML]$Results = $Reader.ReadToEnd()

    $SID = $Results.Response.sid
    Write-Verbose " [New-SplunkSearchJob] :: ID = $SID"

    if($SID)
    {
        if($Realtime)
        {
            while($true)
            {
                Get-SplunkSearchJobResult -SID $SID
                sleep 3
            }
        }
        else
        {
            $cont = $true
            Write-Host "Please wait. It could take a bit..." -NoNewline
            while($cont)
            {
                $Job = Get-SplunkSearchJob -Filter $SID | ?{$_.isDone -eq 1}
                if($Job.ID)
                {
                    $Job | Get-SplunkSearchJobResult 
                    $cont = $false
                    Write-Host
                    Write-Host "Search complete"
                    continue
                }
                sleep 1
            }
        }
    }
}    # New-SplunkSearchJob

iinet
Engager

Any idea where we would find the 'Get-SplunkSearchJob' command?

0 Karma

sideview
SplunkTrust
SplunkTrust

I dont think this gets used or talked about a lot, so mileage may definitely vary.

but note that in the search API, you can supply an 'exec_mode' argument. And if you set it to 'oneshot', it will return the data from the initial POST.

look in this page for exec_mode and oneshot: http://www.splunk.com/base/Documentation/latest/Developer/RESTSearch

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...