When the problem is that "association between my N fields is being lost in my transforming commands", often the answer is to glue the N things together before the stats, and then unpack them later. This is messy. | eval Interesting_Field_Foo =Interesting_Field + "::" + Interesting_Field_Timestamp | fields Field_ID,Interesting_Field_Foo,Field_Name | stats values(*) as * by Field_ID | mvexpand Interesting_Field_Foo | eval Interesting_Field_Foo=split(Interesting_Field_Foo,"::") | eval Interesting_Field=mvindex(Interesting_Field_Foo,0) | eval Interesting_Field_Timestamp = mvindex However if you were actually interested in more than one field value from A, this becomes a lot less viable as a solution. What you'll get in your case is fine I think -- one row per value of Interesting_Field, with Field_Name carried along on all those events. Probably fine. However if sourcetype A had 5 fields, and many events per Field_Id.... this would be pretty horrible - the mvexpand would multiply all those out and make an unusable mess... As a further comment, this "glue the N things together before the stats and then split them up after the stats" comes up most often when you need to use the chart command with more than one "group by" field. Including this example just cause it might give you more perspective and help you remember the trick later: eg: if you want to do | chart count over name group subgroup by type well, you can't. chart command only allows a single group by field (boo!) So you have to do this loveliness: | eval nameGroupSubgroup=name + "---" + group + "---" + subgroup | chart count over nameGroupSubgroup by type | eval nameGroupSubgroup=split(nameGroupSubgroup,"---") | eval name=mvindex(nameGroupSubgroup,0) | eval group=mvindex(nameGroupSubgroup,1) | eval subgroup=mvindex(nameGroupSubgroup,2) | fields - nameGroupSubgroup | fields name group subgroup * ... View more

Well, you're far better off learning how to use the eval and stats commands to use join less here, quite possibly to avoid using join here at all. The best thing about join, is how intuitive it seems when you start using it. That's kind of the only good thing though. There are a lot of reasons to avoid join. I won't go into them much but they include pitfalls where your results don't include all the data so the final numbers are wrong and in general join searches will be far slower and you'll be pulling lots of data back to the search head instead of doing work on the indexer nodes. Let's take the first join separately. index_vmturbo sourcetype="vmturbo:vm" | join producer_name max=0 [ search index=vmturbo sourcetype="vmturbo:datastore" | fields datastore_name, entity_id | rename datastore_name as producer_name, entity_id as join_id ] This can certainly be rewritten to be done with just eval and stats. index=vmturbo (sourcetype="vmturbo:vm" OR sourcetype="vmturbo:datastore") | eval producer_name=if(sourcetype="vmturbo:datastore",datastore_name,producer_name) | eval join_id=if(sourcetype="vmturbo:datastore",entity_id,join_id) | stats count by producer_name join_id The second "join" can be rewritten to not use the join command by very similar methods, except.... that I see you're making that search run over all time. Presumably the main overall search is being run with some specific time range like "last 7 days". From this point, there are absolutely still several good options open to still not use join here. Rather than guess which one might be appropriate, I'll wait for a comment back from you giving more details on the explicit timerange here, and then I'll update my answer. UPDATE ---- Alright, if the "groupmember" data does indeed have to represent "all vm's ever", then I recommend just baking this off into something else to give you vastly better performance. The simplest and most versatile, but perhaps a little odd, is to bake the information into a lookup. 1) Create a lookup. Here I'm calling it "all_vms". If you've never done this, you'll have to peruse the Splunk docs around "file based lookups" 2) run this search over all time. index=vmturbo sourcetype="vmturbo:entitygroupmembers" | fields entity_id vm_name | rename entity_id as join_id | outputlookup all_vms 3) set up a scheduled search to run this slightly different search every so often to add new vm's to the list. index=vmturbo sourcetype="vmturbo:entitygroupmembers" | fields entity_id vm_name | rename entity_id as join_id | inputlookup append=t all_vms | dedup entity_id vm_name | outputlookup all_vms At this point you'll have a lookup file with two columns entity_id and vm_name, it'll stay relatively up to date as long as nobody messes with that scheduled search. (Other options are to make an oldschool summary index, and also to raise the ante and make an accelerated data model so you can use tstats ) Anyway, assuming you went with a nice simple (somewhat manual) lookup solution, you can then join the rows from this lookup onto our other search and then re-group everything with stats. index=vmturbo (sourcetype="vmturbo:vm" OR sourcetype="vmturbo:datastore") | eval producer_name=if(sourcetype="vmturbo:datastore",datastore_name,producer_name) | eval join_id=if(sourcetype="vmturbo:datastore",entity_id,join_id) | stats count by producer_name join_id | inputlookup append=t all_vms | stats sum(count) as count by producer_name join_id vm_name Notes: The real nitty gritty around the various fields and isn't really specified here, which is fine. But just note that in the last stats command you may actually need something a little different stats values(producer_name) values(vm_name) by join_id Or there may well be other fields join was carrying along that are actually important, that we need to now explicitly pass through in our stats expressions. For further reading, I gave a talk at conf2016 called "Let Stats Sort it Out", the slides for which you can find on the following page under "march 2015" http://wiki.splunk.com/Virtual_.conf Some of the examples are pretty advanced, and it assumes a fairly deep familiarity with eval and stats, but with all its examples you may find it useful. ... View more

There are other answers you might look at, for instance -- https://answers.splunk.com/answers/30678/append-and-max-results-50000.html However the best thing is probably for you to post the syntax of your search and we can see if we can rewrite it to use faster and better SPL without the limits of append/join. ... View more

I have found the root cause of the issue, and it is a mistake in Splunk's code. A file that Splunk's JSChart imports after module initialization accidentally clobbers a bunch of other libraries that other Splunk code had imported earlier. There are three options to fix this. 1) Hotfix Splunk's code directly, 2) If you're already using Sideview Utils, download the latest release which went up yesterday (mostly for this fix), or 3) wait for Splunk to hopefully fix it in a 6.5.X maintenance release. 1) If you need to hotfix a particular build, the following will fix the root cause directly. Open $SPLUNK_HOME/share/splunk/search_mrsparkle/exposed/build/jscharting/index.js the file is minified (even if you have minify_js=False in web.conf) but fortunately the code in question is quite near the beginning . Find the lines with these chars: __WEBPACK_AMD_DEFINE_RESULT__=function(require){return splunk=__webpack_require__("shim/splunk") the "splunk" reference there, is the mistake. As an undeclared variable the JS interpreter will look up the chain and assume the developer is referring to "window.splunk". End result is that window.splunk gets overwritten, including various libraries that had been imported into namespaces therein. (here the particular one was of course splunk.time.DateTime) you can hotfix this by changing return splunk= to something as simple as return splunk2= that is still not great code, but it's a sufficient change to fix the bug. NOTE that you have to explicitly clear the entire browser cache or hit the "bump" endpoint. (hitting reload and even shift-reload will not suffice here. Likewise restarting splunk has no effect whatsoever on clearing browser cache) 2) Dowload Sideview Utils 3.3.16 I was able to ship a patch for this problem. This was actually root cause for two other paying customers with similar cases in the field. I may refine the patch a bit more as time goes on -- I suspect there are quite a few edge case bugs that have as their root cause this same mistake in jscharting/index.js, and if there are others with different manifestations, the particular patch I've done may not suffice to fix them all. 3) Last but not least, Splunk will hopefully fix it themselves now that it has been reported, if you can wait that long. ... View more

You don't actually need streamstats' reset argument to accomplish this, and I think it may be more powerful to attack the problem with a more generic approach using just.... streamstats! o_O. So streamstats will do its count arithmetic split out by any 'split by' field you give it. And at a high level, your problem in this case is that you need it to sort of do it's arithmetic not just by user but also according to this extra reset logic. Solution - -- we can construct that extra reset logic as a "reset_count" field that we build with a separate streamstats command. For each user it will be an integer counting the number of times they have "reset" by getting a success. -- we can then pass that second split-by field as well, to the main streamstats. -- it will then happily do its normal bucketing, grouping by the unique combination of both user and that user's "resetCount" It will look something like this. I recommend getting to know it pipe by pipe to closely understand how it's working, (and to iron out any wrinkles I might have here. Note - you may want to use current=f on the second streamstats - i'm not sure. (It depends on whether you want to count events since the reset, or events since and including the reset) index=* login user=* (result="Success" OR result="Failed") | reverse | streamstats count(eval(result=="Success")) as reset_count by user | streamstats count by user reset_count | reverse And in your _internal example, a working example I was testing with is here: index=_internal component=* | streamstats count(eval(log_level=="WARN")) as reset_count by component | streamstats current=f count by component reset_count | table _time component reset_count log_level count BONUS: As an alternate strategy, which may be preferable if you prefer explicit eval statements to the complex eval() syntax in streamstats, you can construct an explicit marker field using a separate eval field. that would look something like this: | eval is_reset=if(result=="Success",1,null()) | streamstats count(is_reset) as reset_count by user In general I find any of the nested eval() syntax in stats/eventstats/streamstats can always be deconstructed as a more explicit and more readable eval clause plus a stats/eventstats/streamstats without the eval(). But some people also prefer the more compact and more "inscrutable black box" feeling of the nested eval(). ... View more