Using basesearch with subsearch

[fatsug]

I nabbed some searches from our license server/monitoring console and placed them in the search head cluster so that they would be available to some users which should not have access to the monitoring console.

The resulting dashboard overview would benefit (heavily) from a "base search" to feed the different panels. However, some of them use "subsearches" and I cannot figure out if i can and then how to combine the two.

There are a couple of these searches where you pull some license usage data and available license for different pools or the total license available (hence using the stacksz when checking "all" pools ("*")).

index=_internal source=*license_usage.log* type="RolloverSummary" pool=$pool$
| bin _time span=1d
| stats latest(b) AS b by slave, pool, _time
| timechart span=1d sum(b) AS "volume" by pool fixedrange=false
| join type=outer _time [
search index=_internal source=*license_usage.log* type="RolloverSummary" pool=$pool$
| bin _time span=1d
| dedup _time stack
| eval licenzz=if("$pool$"=="*", stacksz, poolsz)
| stats latest(licenzz) AS "Available license" by _time
]
| fields - Temp
| foreach * [
eval <<FIELD>>=round('<<FIELD>>'/1024/1024/1024, 3)
]

Different panels use different "stats" and "evals", different "AS" naming and more. There is however one consistent part, the initial search:

index=_internal source=*license_usage.log* type="RolloverSummary" pool=$pool$

I figured it would be a good ide to use a base search with this, though I cannot figure out how.

Using a larger search including the join and subsearch "sort of works". But getting all the different "stats", "evals" and "AS" to produce the expected output is a nightmare.

The initial and smaller base search above is the smallest common denominator. But then I cant figure out how to reference this base in the subsearch for the join?

All suggestions are welcome. All the best

[PickleRick]

As a rule of thumb, the base search should be a transforming search (i.e. containing stats command or timechart). You can get away with non-transforming search but you should explicitly list the fields which you want to retain from your base search for later use by postprocess searches.

And you definitely don't want too much data returned from the base search (a SH will have to keep this result set for post-processing after all).

So it kinda depends on your whole picture because that's not always about the common denominator.

For example if you have one search

index=a | stats count by fieldb

and another one

index=a | stats count by fieldc

The best base search would be not

index=a | fields fieldb fieldc

But rather

index=a | stats count by fieldb fieldc

And your post-process searches would just do

| stats sum(count) by fieldb

and

| stats sum(count) by fieldc

respectively.

[fatsug]

That is a really great explanation, thank you!

In other words, there would be little to no gain by using my suggested base search as it it would retain a lot of excess data from entire events.

What I could, in theory, do would be to run a basesearch keeping only the 3-4 fields all subsequent panels would use. This would however put a strain on the SH cluster.

So for any real gain here, I would need to rewrite all panels that could use an effective base search to work with something like calculated daily averages and process these for each panel.

However, and I'm sorry for being a stickler, this does not really answer the question regarding using a base search with subserches.

I can run the base search and have a panel use that base with a query. But can you reference a base search withing the query using the base search?

The example below is pretty crappy but hopefully a bit clearer then in my initial post?

<search id="base_search">
    <query>
        index="_internal" | stats count by <something>
    </query>
</search>
...
...
<search base="base_search">
    <query>
        search <field>=<value> 
        | join type=outer _time [ 
            <search referencing the same base_search>
            | stats count something
        ]
    </query>
</search>

In a search which uses a base search (an effective one 😉 ) can I reference the same (or another ) base search inside a "subsearch"/"nested search"?

[PickleRick · Accepted solution]

I don't think so. Post-process search is a parameter for the POST request and needs a valid SPL search. If you wanted to have the post-process search reference the base search itself you'd have to loadjob with that particular search's ID.

EDIT: OK, you can do that using the same saved search (but for this you need a scheduled saved search).

[fatsug]

So in theory doable but practically ridiculous, gotcha 🙂