Archive
Highlighted

Streaming and non streaming commands

Communicator

Can someone explain exact difference between streaming and non-streaming commands in laymen terms?

Thanks

Highlighted

Re: Streaming and non streaming commands

Influencer

The documentation here is pretty clear: http://docs.splunk.com/Documentation/Splunk/6.4.1/Search/Typesofcommands

Can you be more specific on which parts you don't understand?

0 Karma
Highlighted

Re: Streaming and non streaming commands

Esteemed Legend

It really doesn't make any difference, except for a few commands that you might like to use that require it (e.g. multireport). I really never think about this distinction.

0 Karma
Highlighted

Re: Streaming and non streaming commands

Ultra Champion

The following explains it nicely - How to create custom search commands using Splunk SDK for Python

It says -

-- There are two subtypes of custom search commands:
• A streaming custom search command is one to which data is streamed. You can think of it as applying a "function"/"transformation" to each event and then writing out the result of that operation. It is a kind of mapper. An example of such a command might be a command that adds a field to each event.
• A non-streaming custom search command expects to have all the data before it operates on it. As such, it is usually "reducing" the data into the output by applying some sort of summary transformation on it. An example of a non-streaming command is the stats command, which will collect all the data before it can calculate the statistics.

I see a correlation to the hadoop world - the streaming custom search command is something like the map phase of the MapR job while a non-streaming custom search command is the reduce part.

The following streaming command says

-- A command that runs on the indexer and can be applied to subsets of index data in a parallel manner. A streaming command applies a transformation to each event returned by a search. For example, the rex command is streaming because it extracts and adds fields to events at search time.

-- In contrast, non-streaming commands are centralized and run at the search head. These commands analyze the entire set of data available at the search head, and then derive the search result output from that set.

View solution in original post