Streaming and non streaming commands

[splunkn]

Can someone explain exact difference between streaming and non-streaming commands in laymen terms?

Thanks

[ddrillic · Accepted solution]

The following explains it nicely - How to create custom search commands using Splunk SDK for Python

It says -

-- There are two subtypes of custom search commands:
• A streaming custom search command is one to which data is streamed. You can think of it as applying a "function"/"transformation" to each event and then writing out the result of that operation. It is a kind of mapper. An example of such a command might be a command that adds a field to each event.
• A non-streaming custom search command expects to have all the data before it operates on it. As such, it is usually "reducing" the data into the output by applying some sort of summary transformation on it. An example of a non-streaming command is the stats command, which will collect all the data before it can calculate the statistics.

I see a correlation to the hadoop world - the streaming custom search command is something like the map phase of the MapR job while a non-streaming custom search command is the reduce part.

The following streaming command says

-- A command that runs on the indexer and can be applied to subsets of index data in a parallel manner. A streaming command applies a transformation to each event returned by a search. For example, the rex command is streaming because it extracts and adds fields to events at search time.

-- In contrast, non-streaming commands are centralized and run at the search head. These commands analyze the entire set of data available at the search head, and then derive the search result output from that set.

[woodcock]

It really doesn't make any difference, except for a few commands that you might like to use that require it (e.g. multireport). I really never think about this distinction.

[jplumsdaine22]

The documentation here is pretty clear: http://docs.splunk.com/Documentation/Splunk/6.4.1/Search/Typesofcommands

Can you be more specific on which parts you don't understand?