To give further examples, a distributable streaming command that can run on an indexer can also run on the search head, so take this example index=_audit ``` This eval runs on the indexer ``` | eval isAdmin=if(user="admin", 1, 0) ``` This lookup runs on the indexer ``` | lookup actions.csv action OUTPUT action_name ``` This stats runs on both indexer and search head, i.e. the indexer will generate stats and then pass its set of stats to the search head, along with all other stats from other indexers and then the final counters are merged on the search head ``` | stats count by user action_name isAdmin ``` This lookup runs on the search head, as the data now exists on the SH. Once the data is on the SH, it will not go back to the indexer. ``` | lookup users.csv user OUTPUT user_name ``` So now this eval runs on the search head ``` | eval do_alert=if(isAdmin, 1, 0) As you can see it contains some eval, lookup and stats commands. This search will be sent from the SH to the "search peers", which are the indexers it can use to search against. Each indexer will run this same search on the set of data it owns. The key point here is that once it hits the stats command, that is the trigger for the indexers to return their dataset to the search head. If you look at the job properties of any search that does a stats command, you will see in the phase0 detail something like the following for a simple "index=_audit | stats count by user" litsearch index=_audit | addinfo type=count label=prereport_events track_fieldmeta_events=true | fields keepcolorder=t "prestats_reserved_*" "psrsvd_*" "user" | prestats count by user this is showing that the indexer will return some "prestats", which is its own reduced data set that it will send to the search head. In the above example, the first lookup will run first on the indexer then the second on the SH. So when it talks about 'invoking' the command, it's really about where the data happens to be in the execution of the entire SPL.  As you can see, as soon as you use a dataset processing command or a transforming command, the data is shifted from the indexers to the search head, so you immediately lose parallelism, so it is best to put those type of commands as far down the SPL pipeline as possible. If you look at the command types table, you can see some commands can work differently depending on how it's called, e.g. fillnull is a dataset processing command with no parameters, but distributable streaming when used with a field name, so be aware of these subtle distinctions when considering search performance.

Oh dang, good catch with the trailing comma! So just tried to limit the initially called out events as much as possible on the base search with the additional filter of  (disposition.disposition="TERMINATED" OR "connections{}.left.facets{}.number"=*) and then limiting the stats aggregations to just the fields that are required for your downstream analysis and display. Glad its running faster!

Ok, i had to make some minor changes, things I left out of my original json to simplify, but YES this did run faster! What is the reason for it tho? Seems like it avoids the (costly?) | stats values(*) as * by guid or passing in the guids from a subsearch. I'll have to stare at that 'max(eval)' statement for awhile to get it! Here is my updated SPL and changed i made (some renaming and mvdedup for multivalue field stuff and removed a trailing comma you had after guid_terminated=1)   index="my_data" resourceId="enum*" (disposition.disposition="TERMINATED" OR "connections{}.left.facets{}.number"=*) | rename "connections{}.left.facets{}.number" as sourcenumber | eval sourcenumber=mvdedup(sourcenumber) | rename disposition.disposition as disposition | stats max(eval(if(disposition=="TERMINATED", 1, 0))) as guid_terminated, values(sourcenumber) as sourcenumber by guid | where 'guid_terminated'==1 | stats count as count by sourcenumber | sort -count      

Hi @gcusello and @dtburrows3 , Thanks for getting back to me. Sorry if my question wasn't 100% clear. So my current goal is that I'm attempting to create a dashboard. In one panel I have a base search of: index="windows_logs" LogName="Security" Account_Domain=EXCH OR Account_Domain="-" EventCode="4625" OR EventCode="4740" user="john@doe.com" OR user="johndoe" This is to grab the reason an account was locked out and would also show the source IP of that information. I essentially need to grab the IP information from this initial search so I can use it in the follow search: index="iis_logs" sourcetype="iis" s_port="443" sc_status=401 cs_method!="HEAD" c_ip=<source IP information from initial search> I tried to use a subsearch, but being the I am pulling from an index with iis logs, it's too large of a search and times out before it can complete.

Hi @cooldude1812  >>>it looks like all versions of SPlunk (up to 9.1.2) will work on Linux 3.x and 4.x kernels.  Yes, right. in fact, 5x also good (All Linux x86 (64-bit) 3x, 4x and 5x are good with Splunk 9.1.2)   >>>My concern about the 8.2.x is that SPlunk clearly says this is no longer supported since 9/30/23.x Yes, you are right.  Splunk Enterprise version 8.2.x is no longer supported as of September 30, 2023. To have Splunk support and to avoid the security issues on older versions, the Splunk Customers should upgrade from 8.2.x to 9.x.x.  hope you understand the issue better now, thanks. 

Hi some additional comments. ”depending on where in the search the command is invoked” I understand this like the position of command in SPL query define is this command run as normally parallel on indexers or is it run on SH. The key event is, are there any transforming commands positioned in SPL query before streaming command. When the search process returns back to SH it’s never go back to parallel mode to indexers. Example of this is e.g. .... | fields a b | stats values(a) by b vs .... | table a b | stats values(a) by b in first example stats is run on indexers and 2nd one in SH. r. Ismo

So I think on your streamstats example, the first usage of streamstats should be replaced with at "| stats" command instead. So something like this. | inputlookup direct_deposit_changes_v4_1_since_01012020.csv | eval _time=strptime(_time,"%Y-%m-%d") | stats count as daily_count by _time | sort 0 +_time | streamstats window=28 list(daily_count) as values_added_together, sum(daily_count) as sum_daily_count | table _time, daily_count, values_added_together, sum_daily_count  

@dtburrows3  Did I set this up correctly? Note: I posted 5 days to simplify the use case but I need 28-day sums. | inputlookup direct_deposit_changes_v4_1_since_01012020.csv | eval _time = strptime(_time,"%Y-%m-%d") | sort 0 _time | streamstats count as daily_count by _time | streamstats window=28 list(daily_count) as values_added_together, sum(daily_count) as sum_daily_count | table _time, daily_count, values_added_together, sum_daily_count I ended up with over 3 million rows when it should have been around 1,460. Because it wasn't grouped by _time (%Y-%m-%d).   However, the foreach code produced the results I was looking for. | inputlookup direct_deposit_changes_v4_1_since_01012020.csv | eval _time = strptime(_time,"%Y-%m-%d") | stats count as daily_count by _time | mvcombine daily_count | eval cnt=0 | foreach mode=multivalue daily_count [| eval summation_json=if( mvcount(mvindex(daily_count,cnt,cnt+27))==28, mvappend( 'summation_json', json_object( "set", mvindex(daily_count,cnt,cnt+27), "sum", sum(mvindex(daily_count,cnt,cnt+27)) ) ), 'summation_json' ), cnt='cnt'+1 ] | rex field="summation_json" "sum\"\:(?<sum_daily_count>\d+)\}" | fields sum_daily_count | mvexpand sum_daily_count   I confirmed these were correct using Excel. Now I must add _time (%Y-%m-%d) to the results. Thanks and God bless, Genesius

Splunk uses the map/reduce model for searching data.  The search head is the map/reduce coordinator.  The SH sends the query to each indexer for execution against the data stored on that indexer.  Each indexer sends its results back to the SH where they are combined and presented. Some SPL commands, however, must be executed against the full result set and so must be performed by the SH.  When such a command is reached, the indexer stops it portion of the search and returns the current results to the SH.  The SH then completes the query.  For example, the stats command must be performed by the SH.  A distributable streaming command that follows stats will be performed by the SH; otherwise it will be performed by the indexer.

O interesting! I thought I could reference "$data_source.result$" but I see that you need to tokenize it to make the result available. Also I thought there was some kind of automatic binding where an update to one component would be available through a binding to another component that references a token. So I need to use the done handler to create the token, as you show with XML. Unfortunately, my page markup is in JSON and my effort at translation does not work. The JSON will not save: "ds_lo0yYuYR": { "type": "ds.chain", "options": { "query": "| where type=\"SHIP_FROM_STORE\"\n| stats count as sfsCount", "done": { "set": { "token": { "Numerator": "$result.count$" } } }, "extend": "ds_z3a1i32Y", "enableSmartSources": true }, "name": "SFS_Count" },  

I agree with @gcusello here. I did notice the use of the | top limit=1 Source_Network_Address in the original subsearch which I think implies that you are trying to scope the search down to a single IP address that shows up the most often in the windows_logs index and not in the 192.168.0.0/16 range. Which I think can be done with a couple of additional lines like this. (index="iis_logs" sourcetype="iis" s_port="443" sc_status=401 cs_method!="HEAD") OR (index="windows_logs" LogName="Security" Account_Domain=EXCH OR Account_Domain="-" EventCode="4625" OR EventCode="4740" user="john@doe.com" OR user="johndoe") | eval c_ip=coalesce(Source_Network_Address,c_ip) | stats dc(index) AS index_count, count(eval('index'=="windows_logs")) as win_log_count, values(*) AS * BY c_ip | where index_count=2 AND NOT cidrmatch("192.168.0.0/16", c_ip) | sort 1 -win_log_count