topic Re: Store search results in token and form token. in Dashboards & Visualizations

How to store search results in token and form token?

jprovenzale — Wed, 11 May 2022 23:21:55 GMT

Hello,

I have a dashboard that has a multi-select dropdown that contains a list of all database names.

When the dashboard is first run, the token that would hold the database name if a selection was made in the dropdown is set to * so all database events are read. Only the top 5 are returned.

My query looks like this:

index=whatever shard IN ("*")
| chart count as result by shard
| sort -result
| head 5

So say the display panel shows results for these databases.

229, 290, 112, 273, 242

I want to set the dropdown labelled Shards form token "form.shardToken" to the list of databases returned as well as updating the token shardToken with the same list of databases.

Hopefully that all makes sense.

Re: Store search results in token and form token

ITWhisperer — Wed, 11 May 2022 08:01:54 GMT

shardToken will normally be set based on the selection from the dropdown. Do you want the full list of databases stored in another token (which isn't changed by the selection), or do you want the value of * to be the full (or partial / top 5) list?

Re: Store search results in token and form token

jprovenzale — Wed, 11 May 2022 08:10:02 GMT

When the dashboard first runs, it checks all databases and returns the top 5.

I want the list of databases returned set to the dropdown form.shardToken as if they had been selected by the user and the actual token, shardToken, also set to the same returned list.

Re: Store search results in token and form token

ITWhisperer — Wed, 11 May 2022 08:25:52 GMT

Create a search at the top of the dashboard (not associated with a panel)

index=whatever shard IN ("*") | chart count as result by shard | sort -result | head 5 | stats values(shard) as shard

Add this done handler to set the form token based on the results of the search

<done> <eval token="form.shardToken">$result.shard$</eval> </done>

Re: Store search results in token and form token

jprovenzale — Wed, 11 May 2022 23:02:56 GMT

Ok, I've done that but am receiving a warning as shown in the screenshot.

Is "init" a good place for the search and where should the done handler exist?

Thanks

Re: Store search results in token and form token

ITWhisperer — Thu, 12 May 2022 04:24:21 GMT

No, an init block might contain token setting not searches

<form theme="light"> <label></label> <description></description> <search> <query> your search </query> <done> done handling </done> </search> <init> token setting </init> <fieldset submitButton="false" autoRun="false"> input definitions etc.

Re: Store search results in token and form token

jprovenzale — Thu, 12 May 2022 05:21:56 GMT

Thanks very much for your help so far.

I have made the changes and all warnings have cleared.
But no data is returned.

When I run the search in a search window, it DOES return the top 5 shards.
So something in the return assignment is not right. Code looks like this:

<form theme="dark">
<label>Troubleshooting Dashboard</label <description>=======================================================</description> <search>
<query>
index=whatever shard IN ("*")
| chart count as result by shard
| sort -result
| head 5
| stats values(shard) as returnshards
</query>
<done>
<eval token="form.shardToken">$result.returnshards$</eval>
</done>
</search>
<init>
<set token="accountToken">"*"</set>
<set token="accountselectedToken">False</set>
<set token="shardToken">$form.shardToken$</set>
</init>
<fieldset submitButton="true" autoRun="true">

Re: Store search results in token and form token

ITWhisperer — Thu, 12 May 2022 05:37:53 GMT

It is not clear what else might be wrong since you haven't provided the rest of the dashboard. Having said that, try using eval instead of set

<eval token="shardToken">$form.shardToken$</eval>

Re: Store search results in token and form token

jprovenzale — Thu, 12 May 2022 06:12:40 GMT

I attempted to change set to eval with no luck.

I can provide the whole dashboard if you think it would help, but I see the issue in the <done> handler.

The query is fine as I can run it successfully in a search window, so there is a disconnect between the query statement :

| stats values(shard) as returnshards

and

<done>
<eval token="form.shardToken">$result.returnshards$</eval>
</done>

if you need the full dashboard, please let me know.

Thanks

Re: Store search results in token and form token

ITWhisperer — Thu, 12 May 2022 06:17:23 GMT

The full dashboard would be helpful or at least all the relevant parts.

Re: Store search results in token and form token.

jprovenzale — Thu, 12 May 2022 23:23:57 GMT

Ok.

Full dashboard below, with all sensitive bits changed and a couple of panels removed because of 20000 ch limit.

<label>Troubleshooting Dashboard</label>

<query>

index=whatever shard IN ("*") | search shard!=0 AND shard!=-1

| chart count as result by shard

| sort -result

| head 5

| stats values(shard) as returnshards

</query>

<done>

<eval token="form.shardToken">$result.returnshards$</eval>

</done>

</search>

<init>

<set token="accountselectedToken">False</set>

<set token="shardToken">$form.shardToken$</set>

</init>

<label>Shards</label>

<fieldForLabel>shardaccount</fieldForLabel>

<fieldForValue>shard</fieldForValue>

<query>| inputlookup ShardList.csv

| eval shardaccount=shard + " - " + account</query>

</search>

<set token="accountselectedToken">False</set>

</condition>

</condition>

</change>

</input>

<label>Time Range</label>

</default>

</input>

<choice value="1">Show Debug Panel</choice>

</input>

</fieldset>

<row>

<title>Token Debug</title>

<html>

<h2>Shard token = $shardToken$</h2>

<h2>Account token = $accountToken$</h2>

<h2>Account selected token = $accountselectedToken$</h2>

<h2>Debug token = $debugToken$</h2>

<h2>Form token = $form.shardToken$</h2>

<h2>Returned shards = $result.returnshards$</h2>

</html>

</panel>

</row>

<row>

<panel>

<title>Events Count By Shard Descending</title>

<chart>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1

| chart count as result by shard

| sort -result

| head 5

</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<option name="charting.axisTitleX.text">shards</option>

<option name="charting.axisTitleX.visibility">visible</option>

<option name="charting.axisTitleY.text">Event Range</option>

<option name="charting.axisTitleY.visibility">collapsed</option>

<option name="charting.chart">column</option>

<option name="charting.legend.labelStyle.overflowMode">ellipsisEnd</option>

<option name="refresh.display">preview</option>

<set token="shardToken">$click.value$</set>

<set token="form.shardToken">$click.value$</set>

</drilldown>

</chart>

</panel>

<panel>

<title>Accounts</title>

<table>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1

| spath account

| search account=$accountToken$

| where account!=0

| top limit=5 account by shard

| sort - count</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<set token="form.shardToken">$click.value$</set>

<set token="shardToken">$click.value$</set>

<set token="accountToken">$click.value2$</set>

</drilldown>

</table>

</panel>

<panel>

<title>Endpoints</title>

<chart>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1

| chart count by endpoint

| sort - count

| head 5</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<option name="charting.axisTitleX.visibility">collapsed</option>

<option name="charting.axisTitleY.visibility">collapsed</option>

<option name="charting.axisTitleY2.visibility">visible</option>

<option name="charting.chart">column</option>

<option name="charting.chart.stackMode">default</option>

<option name="refresh.display">progressbar</option>

<option name="trellis.size">small</option>

<option name="trellis.splitBy">_aggregation</option>

</chart>

</panel>

</row>

<row>

<panel>

<title>Top 5 External IP Addresses</title>

<chart>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1 NOT (original_ip=“123456789”) NOT (original_ip="-") NOT (original_ip=“123456789”)

| timechart limit=5 count by original_ip useother=f

| sort - count</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<option name="charting.axisTitleX.text">@timestamp by 30 minutes</option>

<option name="refresh.display">progressbar</option>

</chart>

</panel>

<panel>

<title>Top 5 IP External Addresses Word Count</title>

<table>

<query>index=whatever shard IN ($shardToken$) | search shard!=0 AND shard!=-1 NOT (original_ip=“123456789”) NOT (original_ip="-") NOT (original_ip=“123456789”)

| top limit=5 original_ip

| sort - count</query>

<earliest>$TimeRange.earliest$</earliest>

<latest>$TimeRange.latest$</latest>

</search>

<option name="refresh.display">progressbar</option>

</table>

</panel>

</row>

</form>

Re: Store search results in token and form token.

ITWhisperer — Fri, 13 May 2022 12:32:48 GMT

Try removing the assignments to shardToken - there are numerous places where you have done this.

shardToken is the output token from the multiselect input and gets set by the input when you change the input form.shardToken token.

Re: Store search results in token and form token.

jprovenzale — Sat, 14 May 2022 01:51:51 GMT

I don't see how this fixes my problem.

shardToken is only set in 3 places, one from the input dropdown and 2 in drilldowns, which is appropriate.

Re: Store search results in token and form token.

jprovenzale — Tue, 17 May 2022 23:12:49 GMT

So what is wrong with the below code that is not populating the form.shardToken?

<form theme="dark">
<label>Troubleshooting Dashboard</label>
<description>===========================================</description>
<search
<query>
index=whatever shard IN ("*") | search shard!=0 AND shard!=-1
| stats count as result by shard
| sort -result
| head 5
| stats values(shard) as returnshards
</query>
<done>
<eval token="form.shardToken">$result.returnshards$</eval>
</done>
</search>

Re: Store search results in token and form token.

ITWhisperer — Wed, 18 May 2022 04:55:54 GMT

There doesn't appear to be anything wrong with this code snippet so form.shardToken is probably being overwritten elsewhere in your dashboard.

Re: Store search results in token and form token.

jprovenzale — Wed, 18 May 2022 23:49:19 GMT

That may be possible.

There is an initial pre-population of shards into the multi-select dropdown. This could affect the value of form.shardToken.

I guess it depends on what happens first, the initial pre-query or the pre-population.

Judging by current results I'm thinking the query runs first and the form.shardToken is overwritten by the pre-population.

If that's the case, I can store the query result in a temporary token, but where do I set the form.shardToken after the pre-population?
I attempted a done clause as part of the pre-population but this was not allowed and also attempted to assign in the init section.

Thanks

Re: Store search results in token and form token.

ITWhisperer — Thu, 19 May 2022 05:21:06 GMT

What do you mean by pre-population?

Why are you doing "pre-population" when the "pre-query" is doing the job you want?

Re: Store search results in token and form token.

jprovenzale — Thu, 19 May 2022 05:24:29 GMT

I am pre-populating the multi-select with.a list of all available shards.

The pre-query will simply assign the shards to that multi-select that it returns in the result.

In this way, initially, the multi-select will show the shards returned by the pre-query, however dropping the dropdown, they can add any other shards they may want to the list of shards to query.

Re: Store search results in token and form token.

ITWhisperer — Thu, 19 May 2022 05:37:55 GMT

Can you share the code that is "pre-populating" the dropdown?

Re: Store search results in token and form token.

jprovenzale — Mon, 23 May 2022 00:45:55 GMT

It's in the code I gave you earlier but here it is again.

</init>
<fieldset submitButton="true" autoRun="true">
<input type="multiselect" token="shardToken" searchWhenChanged="false">
<label>Shards</label>
<delimiter>,</delimiter>
<fieldForLabel>shardaccount</fieldForLabel>
<fieldForValue>shard</fieldForValue>
<search>
<query>| inputlookup ShardList.csv
| eval shardaccount=shard + " - " + account</query>
<earliest>@d</earliest>
<latest>now</latest>
</search>