What are the known/possible ways of drilling down into a result set without re-querying the entire index?

[rayfoo]

Wanted to see what is/are the possible methods to do so.

One way I could think of is to export the results using outputcsv, then use inputcsv to perform my drill down on that specific result set.

Anyone has better/other methods?

[rayfoo]

The use case I'm thinking of is primarily for interactive searching, where I progressively drill down searches on the previous resultset.

[Lowell · Accepted solution]

Two quick ideas.

1.) If your are doing an interactive search with a massive data set, or a complex search that takes a long time to run (normally when searching over a very large time window), then you can always "save" your search results... Then find the job id from the "jobs" window, and use a search like: | loadjob <job_id> | search <your additional search criteria> and filter it down that way. (Doing the loadjob has a few advantages over loadcsv, such as working with the transforms operator, to give just one example.)

2.) If you are trying to create a dashboard you should look into the HiddenPostProcess module, which lets you run one base search and then put different filters or stats operations on top of the generated results. (There are some known issues with this in Splunk 4.0, or at least using the simple XML variation, if I remember correctly)

Then there is always summary indexing; which requires more setup and planning to get right; but is a very good long-term solution.

[rayfoo]

Thanks, this works, mostly!

Am having some issues related to loadjob though, asked the question at http://answers.splunk.com/questions/1176/is-it-normal-for-the-loadjob-command-in-4-0-10-to-only-load...

[Lowell]

More info would be helpful. For example, are you asking for interactive searching purposes, or are you trying to re-use you search results in a form search or dashboard?

It is possible to this in either way, but what it's all depends on your scenario.

[jrodman]

Not sure what the drill down use cases are.

In 4.1, there are drill downs from reports to subselected datasets. These are fresh searches. However, in many situations, a fresh search against a warm-cache is sufficiently high performance as to not really be a problem. The outputcsv/inputcsv method is simply storing the data in another place in addition to the index.

It depends signifcantly on the expense of the search, and wether the items you wish to drill into are transformed (eg stats etc) as compared to the original, where it might be worth doing this kind of mechanism.

I'm not aware of any built-in mechanism for iteratively refocusing within a dataset without re-running the search. Obviously, summary indexing is along these lines, but it isn't ad-hoc.

It really comes down to the goal. Your method might be the best currently available for what you're trying to do, but it's hard to guess.

[jrodman]

It all depends, but you may be right for your case.

Lowell's clever |loadjob command should get you what you need.

[rayfoo]

Given the goal of drillling down in an interactive search:

I'm thinking that a drilldown on a specific resultset would be faster than even on a warm cache (correct me if I'm wrong please), since the number of raw events in that timeframe is in the order of millions (say about 30mil), and the starting point for drilling down within that timeframe is only about a million or so.