topic Re: Why do my post-process timecharts display "no results found" in dashboard, but the query on its own is fine? in Splunk Search

Why do my post-process timecharts display "no results found" in dashboard, but the query on its own is fine?

dave4988 — Tue, 06 Jun 2017 21:09:38 GMT

I have a simple-xml Splunk dashboard with a base query, and two post-processing queries inheriting from the base. However, when I load the dashboard, it always says "No results found." When I click the "Open in search" button, the results show as expected. Also, when I take out of base search and just throw the entire search into both panels, the charts display as expected. Anyone know what's going on here?

Here's the dashboard xml that isn't working:

<dashboard>
    <label>Test Dashboard</label>
    <description>This is a test</description>
        <search id="base">
            <query>
                index=app sourcetype=tracelog splunk_server_group=prod 
                eventName=business:Logout
                (NOT description="*invalid username or password*")
                NOT code="6703" NOT code="6704" NOT "code=8006" NOT "code=6900" NOT "code=6000" 
            </query>
        </search>
    <row>
        <panel>
            <title>Test chart 1</title>
            <chart>
                <search base="base">
                    <query>
                        search success=false AND agent=true | timechart count by errors
                    </query>
                </search>
                <option name="charting.chart.stackMode">stacked</option>
                <option name="charting.chart">column</option>
            </chart>
        </panel>
    </row>
        <row>
        <panel>
            <title>Test chart 2</title>
            <chart>
                <search base="base">
                    <query>
                        search success=false AND agent=false | timechart count by errors
                    </query>
                </search>
                <option name="charting.chart.stackMode">stacked</option>
                <option name="charting.chart">column</option>
            </chart>
        </panel>
    </row>
</dashboard>

However, if I combine the queries and get rid of the base query as seen below, it works:

<dashboard>
    <label>Test Dashboard</label>
    <description>This is a test</description>
    <row>
        <panel>
            <title>Test chart 1</title>
            <chart>
                <search>
                    <query>
                        index=app sourcetype=tracelog splunk_server_group=prod 
                        eventName=business:Logout
                        (NOT description="*invalid username or password*")
                        NOT code="6703" NOT code="6704" NOT "code=8006" NOT "code=6900" NOT "code=6000" 
                        | search success=false AND agent=true | timechart count by errors
                    </query>
                </search>
                <option name="charting.chart.stackMode">stacked</option>
                <option name="charting.chart">column</option>
            </chart>
        </panel>
    </row>
        <row>
        <panel>
            <title>Test chart 2</title>
            <chart>
                <search>
                    <query>
                        index=app sourcetype=tracelog splunk_server_group=prod 
                        eventName=business:Logout
                        (NOT description="*invalid username or password*")
                        NOT code="6703" NOT code="6704" NOT "code=8006" NOT "code=6900" NOT "code=6000" 
                        |  search success=false AND agent=false | timechart count by errors
                    </query>
                </search>
                <option name="charting.chart.stackMode">stacked</option>
                <option name="charting.chart">column</option>
            </chart>
        </panel>
    </row>
</dashboard>

Any ideas? Am I missing something here?

Re: Why do my post-process timecharts display "no results found" in dashboard, but the query on its own is fine?

gcusello — Wed, 07 Jun 2017 06:38:02 GMT

Hi,
the problem is that a post process search doesn't pass all fields to the related searches, so the following commands fail (in your example "success=false AND agent=true | timechart count by errors") because fields haven't values, instead if you run it as a search both post process and panel search are united and so they run.
To solve the problem, you have to insert at the end of each post process search the command "fields" with the all fields you use in the panels searches.
In your example:

| fields _time, success, agent, errors

Bye.
Giuseppe

Re: Why do my post-process timecharts display "no results found" in dashboard, but the query on its own is fine?

niketn — Wed, 07 Jun 2017 08:01:52 GMT

@dave4988, First off you should start with some Search optimization.
Few applicable in your case are (http://docs.splunk.com/Documentation/Splunk/latest/Search/Writebettersearches and https://docs.splunk.com/Documentation/Splunk/latest/Search/NOTexpressions)
1) Avoid using NOT.
2) Filter results upfront.
3) Inclusion is better than exclusion.

Then you also need to review condition for Post Processing and optimization (http://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches#Post-process_searches_2)
1) Base query should have transforming command
2) Do not pass on raw events
3) Filter required results in the base search

See some of the scenarios with examples for Post Processing with chained and complex post processing.

Having said the above, try out the following query as base search:

index=app sourcetype=tracelog splunk_server_group=prod 
                 eventName=business:Logout
                 ( description!="*invalid username or password*" )
                 ( code!="6703" code!="6704" code!="8006" code!="6900" code!="6000" )
                 success=false AND ( agent="true" OR agent="false" )
| timechart count(eval(agent=="true")) as "True" count(eval(agent=="false")) as "False" by errors

Then in your Post Process search you can use, For agent="true" scenario,

| fields True*
| rename True* as *
| timechart sum(*) as *

For agent="false" scenario,

| fields True*
| rename True* as *
| timechart sum(*) as *