Splunk Search

Why do my post-process timecharts display "no results found" in dashboard, but the query on its own is fine?

dave4988
Engager

I have a simple-xml Splunk dashboard with a base query, and two post-processing queries inheriting from the base. However, when I load the dashboard, it always says "No results found." When I click the "Open in search" button, the results show as expected. Also, when I take out of base search and just throw the entire search into both panels, the charts display as expected. Anyone know what's going on here?

Here's the dashboard xml that isn't working:

<dashboard>
    <label>Test Dashboard</label>
    <description>This is a test</description>
        <search id="base">
            <query>
                index=app sourcetype=tracelog splunk_server_group=prod 
                eventName=business:Logout
                (NOT description="*invalid username or password*")
                NOT code="6703" NOT code="6704" NOT "code=8006" NOT "code=6900" NOT "code=6000" 
            </query>
        </search>
    <row>
        <panel>
            <title>Test chart 1</title>
            <chart>
                <search base="base">
                    <query>
                        search success=false AND agent=true | timechart count by errors
                    </query>
                </search>
                <option name="charting.chart.stackMode">stacked</option>
                <option name="charting.chart">column</option>
            </chart>
        </panel>
    </row>
        <row>
        <panel>
            <title>Test chart 2</title>
            <chart>
                <search base="base">
                    <query>
                        search success=false AND agent=false | timechart count by errors
                    </query>
                </search>
                <option name="charting.chart.stackMode">stacked</option>
                <option name="charting.chart">column</option>
            </chart>
        </panel>
    </row>
</dashboard>

However, if I combine the queries and get rid of the base query as seen below, it works:

<dashboard>
    <label>Test Dashboard</label>
    <description>This is a test</description>
    <row>
        <panel>
            <title>Test chart 1</title>
            <chart>
                <search>
                    <query>
                        index=app sourcetype=tracelog splunk_server_group=prod 
                        eventName=business:Logout
                        (NOT description="*invalid username or password*")
                        NOT code="6703" NOT code="6704" NOT "code=8006" NOT "code=6900" NOT "code=6000" 
                        | search success=false AND agent=true | timechart count by errors
                    </query>
                </search>
                <option name="charting.chart.stackMode">stacked</option>
                <option name="charting.chart">column</option>
            </chart>
        </panel>
    </row>
        <row>
        <panel>
            <title>Test chart 2</title>
            <chart>
                <search>
                    <query>
                        index=app sourcetype=tracelog splunk_server_group=prod 
                        eventName=business:Logout
                        (NOT description="*invalid username or password*")
                        NOT code="6703" NOT code="6704" NOT "code=8006" NOT "code=6900" NOT "code=6000" 
                        |  search success=false AND agent=false | timechart count by errors
                    </query>
                </search>
                <option name="charting.chart.stackMode">stacked</option>
                <option name="charting.chart">column</option>
            </chart>
        </panel>
    </row>
</dashboard>

Any ideas? Am I missing something here?

0 Karma
1 Solution

niketn
Legend

@dave4988, First off you should start with some Search optimization.
Few applicable in your case are (http://docs.splunk.com/Documentation/Splunk/latest/Search/Writebettersearches and https://docs.splunk.com/Documentation/Splunk/latest/Search/NOTexpressions)
1) Avoid using NOT.
2) Filter results upfront.
3) Inclusion is better than exclusion.

Then you also need to review condition for Post Processing and optimization (http://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches#Post-process_searches_2)
1) Base query should have transforming command
2) Do not pass on raw events
3) Filter required results in the base search

See some of the scenarios with examples for Post Processing with chained and complex post processing.

Having said the above, try out the following query as base search:

index=app sourcetype=tracelog splunk_server_group=prod 
                 eventName=business:Logout
                 ( description!="*invalid username or password*" )
                 ( code!="6703" code!="6704" code!="8006" code!="6900" code!="6000" )
                 success=false AND ( agent="true" OR agent="false" )
| timechart count(eval(agent=="true")) as "True" count(eval(agent=="false")) as "False" by errors

Then in your Post Process search you can use, For agent="true" scenario,

| fields True*
| rename True* as *
| timechart sum(*) as *

For agent="false" scenario,

| fields True*
| rename True* as *
| timechart sum(*) as *
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

niketn
Legend

@dave4988, First off you should start with some Search optimization.
Few applicable in your case are (http://docs.splunk.com/Documentation/Splunk/latest/Search/Writebettersearches and https://docs.splunk.com/Documentation/Splunk/latest/Search/NOTexpressions)
1) Avoid using NOT.
2) Filter results upfront.
3) Inclusion is better than exclusion.

Then you also need to review condition for Post Processing and optimization (http://docs.splunk.com/Documentation/Splunk/latest/Viz/Savedsearches#Post-process_searches_2)
1) Base query should have transforming command
2) Do not pass on raw events
3) Filter required results in the base search

See some of the scenarios with examples for Post Processing with chained and complex post processing.

Having said the above, try out the following query as base search:

index=app sourcetype=tracelog splunk_server_group=prod 
                 eventName=business:Logout
                 ( description!="*invalid username or password*" )
                 ( code!="6703" code!="6704" code!="8006" code!="6900" code!="6000" )
                 success=false AND ( agent="true" OR agent="false" )
| timechart count(eval(agent=="true")) as "True" count(eval(agent=="false")) as "False" by errors

Then in your Post Process search you can use, For agent="true" scenario,

| fields True*
| rename True* as *
| timechart sum(*) as *

For agent="false" scenario,

| fields True*
| rename True* as *
| timechart sum(*) as *
____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

gcusello
SplunkTrust
SplunkTrust

Hi,
the problem is that a post process search doesn't pass all fields to the related searches, so the following commands fail (in your example "success=false AND agent=true | timechart count by errors") because fields haven't values, instead if you run it as a search both post process and panel search are united and so they run.
To solve the problem, you have to insert at the end of each post process search the command "fields" with the all fields you use in the panels searches.
In your example:

| fields _time, success, agent, errors

Bye.
Giuseppe

0 Karma
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...