Base Search for Splunk Dashboard

[Priya70]

Hi all, I need a base search for the following dashboard panels.  The dashboard is running very slowly. 

First Panel:
index=software_inventory sourcetype=software:inventory 
| search *time=$time_selector$* ApplicationName=$software_input$ 
| stats dc(endpoint) as TotalEndpoints, dc(SoftwareVersion) as VersionCount by ApplicationName 
| sort - TotalEndpoints

Second Panel: 
index=software_inventory sourcetype=software:inventory 
| search *time=$time_selector$* ApplicationName="$selected_software$" 
| stats dc(endpoint) as EndpointCount by SoftwareVersion 
| sort - EndpointCount

Third Panel:
index=software_inventory sourcetype=software:inventory 
| search *time=$time_selector$* ApplicationName="$selected_software$" 
| stats dc(endpoint) as EndpointCount by SoftwareVersion 
| sort - EndpointCount

Fourth Panel:
index=software_inventory sourcetype=software:inventory 
| search *time=$time_selector$* ApplicationName="$selected_software$" SoftwareVersion="$selected_version$" 
| table endpoint SoftwareVersion

 I tried the following base search, but it doesn't return all the events since its not a transforming search.

index=software_inventory sourcetype=software:inventory 
| fields ApplicationName, SoftwareVersion, endpoint

[ITWhisperer]

Try removing the leading wildcards - they slow things down and actually are probably irrelevant (as they are implied in a search)

[PickleRick]

1. Please don't start multiple threads about the same problem.

2.

search *time=$time_selector$*

is not a valid SPL syntax

3. The typical approach in this type of cases is to try and find a common aggregation which can be refined further down the road in post-processing searches. In your case the approach which could work would be to return dc(endpoint) by applicationname softwareversion and later aggregating those values. You might want to make sure that you can't have multiple versions of the same app on a single endpoint - your results could be wrong then.

[gcusello]

Hi @Priya70 ,

you have to insert in the base search all the common parts of the search and in the fields statement, all the fields to be used in the panels, something like this:

index=software_inventory sourcetype=software:inventory  *time=$time_selector$* (ApplicationName IN ($software_input$, "$selected_software$")
| stats dc(endpoint) AS Endpoints_count BY ApplicationName SoftwareVersion

Then in the four panels, apply the filters your need:

First Panel:
| search ApplicationName=$software_input$ 
| stats dc(endpoint) as TotalEndpoints, dc(SoftwareVersion) as VersionCount by ApplicationName 
| sort - TotalEndpoints

Second Panel: 
| search ApplicationName="$selected_software$" 
| stats dc(endpoint) as EndpointCount by SoftwareVersion 
| sort - EndpointCount

Third Panel:
| search ApplicationName="$selected_software$" 
| stats dc(endpoint) as EndpointCount by SoftwareVersion 
| sort - EndpointCount

Fourth Panel:
| search ApplicationName="$selected_software$" SoftwareVersion="$selected_version$" 
| table endpoint SoftwareVersion

Two things:

don't use the search command after the main search but only in the panels;

I'm not sure that you can use a filter *time=$time_selector$, I think that you have to use a different filter for this  (or these) field(s).

Ciao.

Giuseppe

[tscroggins]

Hi @Priya70,

Can you provide a deidentified sample of the raw events?

How is _time extracted? Does the _time value match the *time value(s) in your raw events?

How are ApplicationName, SoftwareVersion, and endpoint extracted?

If _time and *time have the same values, you can configure your base search to use the time input directly:

<form version="1.1" theme="light">
  <label>Dashboard</label>
  <fieldset submitButton="false">
    <input type="text" token="selected_software">
      <label>Application Name</label>
    </input>
    <input type="time" token="time_selector">
      <label></label>
      <default>
        <earliest>-24h@h</earliest>
        <latest>now</latest>
      </default>
    </input>
  </fieldset>
  <search id="base">
    <query>index=software_inventory sourcetype=software:inventory ApplicationName=$selected_software|s$</query>
    <earliest>$time_selector.earliest$</earliest>
    <latest>$time_selector.latest</latest>
  </search>
  <row>
    <panel>
      <table>
        <search base="base" />
      </table>
    </panel>
  </row>
</form>

You have enough fields in common across the first three searches that you can aggregate using stats in the base search and then aggregate again in the post-processes searches.

Base (id="base"):

index=software_inventory sourcetype=software:inventory ApplicationName=$selected_software|s$
| stats dc(endpoint) as TotalEndpoints by ApplicationName SoftwareVersion

First Panel (base="base"):

| stats sum(TotalEndpoints) as TotalEndpoints dc(SoftwareVersion) as VersionCount by ApplicationName
| sort - TotalEndpoints

Second Panel (base="base"):

| stats sum(TotalEndpoints) as EndpointCount by SoftwareVersion
| sort - EndpointCount

The third panel is the same as the second.

The fourth panel is just a table of endpoint values for the selected software and version.

If you're lucky, your raw data is structured in a way that will allow you to use tstats. If not, you can use summary indexing or an accelerated data model to make aggregations significantly faster. You previously mentioned you do not want to use scheduled searches, but scheduled summary searches may solve your problem.

[Priya70]

@gcusello @ITWhisperer @tscroggins @PickleRick Thank you for your response. It sort of works.

Why does the following work when I put the token in base search, but not in the panel. I get no results/inaccurate results.

Base:
index=software_inventory sourcetype=software:inventory ApplicationName="$software_input$"
| stats dc(endpoint) AS Endpoints_count BY ApplicationName SoftwareVersion

Panel:
|fields ApplicationName Endpoints_count
|sort 0 Endpoints_count

 But, the following, doesn't work, when I put the token in panel and not in the base search.

Base
index=software_inventory sourcetype=software:inventory
| stats dc(endpoint) AS Endpoints_count BY ApplicationName SoftwareVersion

Panel:
|search ApplicationName="$software_input$
|fields ApplicationName Endpoints_count
|sort 0 Endpoints_count

 The reason I dont want to put token in the base search is because I dont want base search to rerun when a new software is selected. That just defeats the purpose of a base search.

[gcusello]

Hi @Priya70 ,

i suppose that the missing quotes in the panel search is a mistyping, otherwise this is the issue.

Ciao.

Giuseppe.

[tscroggins]

Hi @Priya70,

Have you verified the panel search references the base search? As in the example above, the search element should have a base attribute with the same value as the id attribute of the base search:

The |s in $token|s$ is a token filter. The |s filter sanitizes the token as a quote-enclosed string. Apart from string validation, it helps prevent SPL injection attacks and inadvertent mistakes.

For example, a token named foo with the value bar"baz would produce the following searches:

index=main foo=$foo$
=>
index=main foo=bar"baz

index=main foo="$foo$"
=>
index=main foo="bar"baz"

index=main foo=$foo|s$
=>
index=main foo="bar\"baz"

The first two searches are malformed and generate an "Unbalanced quotes" error.

The third search is well-formed.

The |s filter also prevents foo values like bar" OR baz="qux and bar" | delete | eval baz="qux from being executed:

index=main foo="$foo$"
=>
index=main foo="bar" OR baz="qux"

=>
index=main foo="bar" | delete | eval baz="qux"

index=main foo=$foo|s$
=>
index=main foo="bar\" OR baz=\"qux"

=>
index=main foo="bar\" | delete | eval baz=\"qux"

[Priya70]

Yes, base search is being referenced properly. The weird thing is that it works when the token is applied in the base search, but doesn't work when I remove it from the base search and apply in the panel using where and search commands. 

[tscroggins]

Hi @Priya70,

Without seeing your dashboard source code, I'm not sure there's more we can recommend. Everyone's addressed the most common causes of issues.

To look for specific causes, you can open the problem panel's search in a separate tab to evaluate the generated SPL by clicking the Open in Search (magnifying glass) link. You can inspect search logs for warnings and errors by clicking the Inspect (lowercase "i") link.

[ITWhisperer]

Try using token filters

|search ApplicationName=$software_input|s$

[Priya70]

@ITWhisperer This one doesnt work. BTW, I forgot to end the token in quotation mark in the second example above. The first example is giving me accurate results. The second one just doesn't work.

[ITWhisperer]

Try with where

| where ApplicationName=$software_input|s$

[Priya70]

I've tried where as well, still no luck.