Stop Processing Long Query When No Results Found

[atari1050]

Hello Splunk Gurus-

We have noticed that a Splunk job does not end gracefully (version 6.6.3) if the post-pipe commands encounter missing fields, specifically in the map command.

example:
index="_internal" sourcetype="scheduler" status=skipped savedsearch_name!="_ACCELERATE*" "Your maximum number of concurrent searches has been reached."
| eval time=strftime(_time, "%H:%M:%S")
| stats count, values(time) as times, values(reason) as reasons, by user, savedsearch_id, savedsearch_name
| map search="localop | rest /services/authentication/users/$user$ | fields email | eval user=$user$, savedsearch=$savedsearch_name$, count=$count$, times=$times$, reasons=$reasons$, ssid=$savedsearch_id$"
| rex field=ssid ";(?[^;]+)"
| fields app, savedsearch, reasons, count, times, email

The map command is expecting the user variable to be populated with data. If the specific "Your maximum number..." is not present, thus the user field is not present. Instead of just completing with "No results found", it throws a nasty error and shows as a failure. Of course, we are searching and alerting on failed searches, so this gets annoying.

Is there an eval to NULL we could perform to prevent this from failing and simply present "No results found"?

On a larger scale, is it possible, in general for Splunk to have some Pre- or Post-pipe logic to actually stop the job if no results are encountered before a large amount of processing is done, thus saving CPU and searchhead resources?

Thanks,
Mike

[atari1050]

Fantastic niketnilay, thanks!

[niketn · Accepted solution]

@atari1050, Splunk has depends and rejects attribute to show/hide inputs/visualization elements like row, panel, chart etc. based on whether a token is set or not. You can refer to Splunk Dashboard Examples app (https://splunkbase.splunk.com/app/1603/) which has Null Search Swapper example for this scenario.

For your use case I have added three Search Event Handlers:

1) <done> : When search completes but returns no results, by un-setting the token $displayTable$, custom HTML error message is displayed using rejects attribute
2) <fail> : When search fails, by un-setting the token $displayTable$, custom HTML error message is displayed using rejects attribute
3) <error> : When search errors out, by un-setting the token $displayTable$, custom HTML error message is displayed using rejects attribute

PS: The table will be displayed only when token $displayTable$ is set i.e. search completes and returns results (using <done> Search Event Handler)

<dashboard>
  <label>No Results Found</label>
  <row>
    <panel>
      <!-- Using depends to show table only when displayTable token is set -->
      <table depends="$displayTable$">
        <search>
          <query>index="_internal" sourcetype="scheduler" status=skipped savedsearch_name!="_ACCELERATE*" "Your maximum number of concurrent searches has been reached." | eval time=strftime(_time, "%H:%M:%S") | stats count, values(time) as times, values(reason) as reasons, by user, savedsearch_id, savedsearch_name | map search="localop | rest /services/authentication/users/$user$ | fields email | eval user=$user$, savedsearch=$savedsearch_name$, count=$count$, times=$times$, reasons=$reasons$, ssid=$savedsearch_id$" | rex field=ssid ";(?[^;]+)" | fields app, savedsearch, reasons, count, times, email
          </query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <done>
            <!-- When search returns no results unset token to hide table -->
            <condition match="$job.resultCount$==0">
              <unset token="displayTable"></unset>
            </condition>
            <condition>
              <!-- Set token to display table -->
              <set token="displayTable">true</set>
            </condition>            
          </done>
          <error>
            <!-- In case of search error unset token to hide table -->
            <unset token="displayTable"></unset>
          </error>
          <fail>
            <!-- In case of search failure unset token to hide table -->
            <unset token="displayTable"></unset>
          </fail>          
        </search>
      </table>
      <!-- Using rejects to show Custom Message when displayTable token is unset i.e. No Search Result, Search Error or Search Failure -->
      <html rejects="$displayTable$">
        <div style="font-size:150%;font-weight:bold;color:red;text-align:center">
          Search returned no results. Please consider changing search filters!
        </div>
      </html>
    </panel>
  </row>
</dashboard>

Refer to Splunk Documentation for Search Event Handlers, set and unset tokens,depends and rejects:
http://docs.splunk.com/Documentation/Splunk/latest/Viz/EventHandlerReference#Search_event_handlers
http://docs.splunk.com/Documentation/Splunk/latest/Viz/EventHandlerReference#unset