I often run into a case where I find I need to take the same dataset and compute aggregate statistics on different group-by sets, for instance if you want the output of this:     index=example | stats avg(field1) by x,y,z | append [ index=example | stats perc95(field2) by a,b,c ]   I am using the case n=2 groupbys for convenience. In the general case there are N groupbys, and arbitrary stats functions... what is the best way to optimize this kind of query, without using append (which runs into subsearch limits)? Some of the patterns I can think of are below. One way is to use appendpipe.    index=example | appendpipe [ | stats avg(field1) by x,y,z ] | appendpipe [ | stats perc95(field2) by a,b,c ]   Unfortunately this seems kind of slow, especially once you start having to add more subsearches and preserving and passing  a large number of non-transformed events throughout the search. Another way is to use eventstats to preserve the events data, finishing it off with a final stats.   index=example | eventstats avg(field1) as avg_field1 by x,y,z | stats first(avg_field1) as avg_field1, perc95(field2) by a,b,c   Unfortunately this is not much faster. I think there is another way using streamstats in place of eventstats, but I still haven't figured out how to retrieve the last event without just invoking eventstats last() or relying on an expensive sort.   Another way I've tried is intentionally duplicating your data using mvexpand which has the best performance by far.    index=example ```Duplicate all the data``` | eval key="1,2" | makemv delim="," key | mvexpand key ```Set groupby = concatenation of groupby field values``` | eval groupby=case(key=1,x.",".y.",".z, key2=a.",".b.",".c, true(), null()) | stats avg(field1), perc95(field2) by groupby   Are there any other patterns that are easier/faster? I'm curious as to how Splunk processes things under the hood, I know something called "map-reduce" is part of it but would be curious to know if anyone knows how to optimize this computation and why it's optimal in a theoretical sense.  ... View more

Hello, We have an application pulling search results from a scheduled search using Splunk API periodically, but encountering an issue where there is an excess of expired jobs (5000+) which are being kept for 1 month+ for some reason. Because the application has to look through each of these jobs it's taking too long and timing out.  We tried deleting the expired jobs through the UI but they keep popping back up/not going away. Some of these now say "Invalid SID" when I try to inspect them. Is there any way we can clear these bulk, preferable without resorting to UI (which only shows 50 at a time)?  ... View more

I am running a map command off of an initial search. The map ends with a sendemail command which sends a table of results. I would like to send a message that computes totals and other stats on this table -- however, I would not like to include this data as a totals row the table/search results, only in the message. In other words, the whole email would look something like: Subject: Alert condition triggered Sum(Field 1) of type X results: 524 Table of results ----------------------------------------------- | Field 1 | Field 2 | ... ... ... ... I know this can be done by running yet another subsearch for the "message" parameter in Splunk. However, this means I'm effectively running the same search twice... when performance-wise it would be better to just run the stats off of the table after it is generated. I know how to implement this in a dashboard with base searches, but I would like to know how to do this in 1 search. I think the problem is that there is no "scope" outside of the search results to which I can write a variable. I can think of a clunky solution using lookup/outputlookup. Is there some way to maybe pipe the table into a separate subsearch that generates a variable/token but does not actually append to the main search? ... View more

Hi guys, I am trying to chart multiple days on the same line chart, kind of like in this example (https://docs.splunk.com/Documentation/Splunk/7.3.2/Search/Comparehourlysumsmultipledays) . However I am plotting 5m intervals and not hours like in the example above: ... | bin span = 5m _time | eval clock=strftime(_time,"%H:%M:%S") | chart avg(Total_ct) by clock,day However, the x-axis labels aren't showing up because there are too many (288 of them). What I want is to have 5m data (and thus 288 intervals over the whole day) but still be able to see some labels on the x-axis. This is what Splunk is trying to show and failing, because of the pixel limit: [ 00:00 00:05 00:10 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23:55 ] I would be okay if the x-axis looked like the following: [ 0 1 . . . . . 23] Note: it doesn't have to be an hourly interval, I just want some kind of labels to show up on the x-axis. As it is now, there are no labels and its really hard to tell at a glance what the times are unless I mouse over. I did find a hacky way to get leverage the smart time-scaling display using timechart while also keeping the "chart by time over day" effect. This actually gives me an hourly scale with all 5m time interval data, but when I mouse over each point I see "1971-01-01" on each time (I basically converted all the days into one ridiculous day so I could overlay them). ... | eval clock=strftime(_time,"%H:%M:%S") | eval reclock="1971-01-01"." ".'clock' | eval day=strftime(_time,"%D") | eval _time=strptime(reclock, "%Y-%m-%d %H:%M:%S") | timechart span=5m cont=f avg(Total_ct) by day limit=0 Is there some way I can have my cake and eat it too? ... View more