Getting Data In

Stop Processing Long Query When No Results Found

atari1050
Path Finder

Hello Splunk Gurus-

We have noticed that a Splunk job does not end gracefully (version 6.6.3) if the post-pipe commands encounter missing fields, specifically in the map command.

example:
index="_internal" sourcetype="scheduler" status=skipped savedsearch_name!="_ACCELERATE*" "Your maximum number of concurrent searches has been reached."
| eval time=strftime(_time, "%H:%M:%S")
| stats count, values(time) as times, values(reason) as reasons, by user, savedsearch_id, savedsearch_name
| map search="localop | rest /services/authentication/users/$user$ | fields email | eval user=$user$, savedsearch=$savedsearch_name$, count=$count$, times=$times$, reasons=$reasons$, ssid=$savedsearch_id$"
| rex field=ssid ";(?[^;]+)"
| fields app, savedsearch, reasons, count, times, email

The map command is expecting the user variable to be populated with data. If the specific "Your maximum number..." is not present, thus the user field is not present. Instead of just completing with "No results found", it throws a nasty error and shows as a failure. Of course, we are searching and alerting on failed searches, so this gets annoying.

Is there an eval to NULL we could perform to prevent this from failing and simply present "No results found"?

On a larger scale, is it possible, in general for Splunk to have some Pre- or Post-pipe logic to actually stop the job if no results are encountered before a large amount of processing is done, thus saving CPU and searchhead resources?

Thanks,
Mike

0 Karma
1 Solution

niketn
Legend

@atari1050, Splunk has depends and rejects attribute to show/hide inputs/visualization elements like row, panel, chart etc. based on whether a token is set or not. You can refer to Splunk Dashboard Examples app (https://splunkbase.splunk.com/app/1603/) which has Null Search Swapper example for this scenario.

For your use case I have added three Search Event Handlers:

1) <done> : When search completes but returns no results, by un-setting the token $displayTable$, custom HTML error message is displayed using rejects attribute
2) <fail> : When search fails, by un-setting the token $displayTable$, custom HTML error message is displayed using rejects attribute
3) <error> : When search errors out, by un-setting the token $displayTable$, custom HTML error message is displayed using rejects attribute

PS: The table will be displayed only when token $displayTable$ is set i.e. search completes and returns results (using <done> Search Event Handler)

<dashboard>
  <label>No Results Found</label>
  <row>
    <panel>
      <!-- Using depends to show table only when displayTable token is set -->
      <table depends="$displayTable$">
        <search>
          <query>index="_internal" sourcetype="scheduler" status=skipped savedsearch_name!="_ACCELERATE*" "Your maximum number of concurrent searches has been reached." | eval time=strftime(_time, "%H:%M:%S") | stats count, values(time) as times, values(reason) as reasons, by user, savedsearch_id, savedsearch_name | map search="localop | rest /services/authentication/users/$user$ | fields email | eval user=$user$, savedsearch=$savedsearch_name$, count=$count$, times=$times$, reasons=$reasons$, ssid=$savedsearch_id$" | rex field=ssid ";(?[^;]+)" | fields app, savedsearch, reasons, count, times, email
          </query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <done>
            <!-- When search returns no results unset token to hide table -->
            <condition match="$job.resultCount$==0">
              <unset token="displayTable"></unset>
            </condition>
            <condition>
              <!-- Set token to display table -->
              <set token="displayTable">true</set>
            </condition>            
          </done>
          <error>
            <!-- In case of search error unset token to hide table -->
            <unset token="displayTable"></unset>
          </error>
          <fail>
            <!-- In case of search failure unset token to hide table -->
            <unset token="displayTable"></unset>
          </fail>          
        </search>
      </table>
      <!-- Using rejects to show Custom Message when displayTable token is unset i.e. No Search Result, Search Error or Search Failure -->
      <html rejects="$displayTable$">
        <div style="font-size:150%;font-weight:bold;color:red;text-align:center">
          Search returned no results. Please consider changing search filters!
        </div>
      </html>
    </panel>
  </row>
</dashboard>

Refer to Splunk Documentation for Search Event Handlers, set and unset tokens,depends and rejects:
http://docs.splunk.com/Documentation/Splunk/latest/Viz/EventHandlerReference#Search_event_handlers
http://docs.splunk.com/Documentation/Splunk/latest/Viz/EventHandlerReference#unset

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"

View solution in original post

atari1050
Path Finder

Fantastic niketnilay, thanks!

0 Karma

niketn
Legend

@atari1050, Splunk has depends and rejects attribute to show/hide inputs/visualization elements like row, panel, chart etc. based on whether a token is set or not. You can refer to Splunk Dashboard Examples app (https://splunkbase.splunk.com/app/1603/) which has Null Search Swapper example for this scenario.

For your use case I have added three Search Event Handlers:

1) <done> : When search completes but returns no results, by un-setting the token $displayTable$, custom HTML error message is displayed using rejects attribute
2) <fail> : When search fails, by un-setting the token $displayTable$, custom HTML error message is displayed using rejects attribute
3) <error> : When search errors out, by un-setting the token $displayTable$, custom HTML error message is displayed using rejects attribute

PS: The table will be displayed only when token $displayTable$ is set i.e. search completes and returns results (using <done> Search Event Handler)

<dashboard>
  <label>No Results Found</label>
  <row>
    <panel>
      <!-- Using depends to show table only when displayTable token is set -->
      <table depends="$displayTable$">
        <search>
          <query>index="_internal" sourcetype="scheduler" status=skipped savedsearch_name!="_ACCELERATE*" "Your maximum number of concurrent searches has been reached." | eval time=strftime(_time, "%H:%M:%S") | stats count, values(time) as times, values(reason) as reasons, by user, savedsearch_id, savedsearch_name | map search="localop | rest /services/authentication/users/$user$ | fields email | eval user=$user$, savedsearch=$savedsearch_name$, count=$count$, times=$times$, reasons=$reasons$, ssid=$savedsearch_id$" | rex field=ssid ";(?[^;]+)" | fields app, savedsearch, reasons, count, times, email
          </query>
          <earliest>-24h@h</earliest>
          <latest>now</latest>
          <done>
            <!-- When search returns no results unset token to hide table -->
            <condition match="$job.resultCount$==0">
              <unset token="displayTable"></unset>
            </condition>
            <condition>
              <!-- Set token to display table -->
              <set token="displayTable">true</set>
            </condition>            
          </done>
          <error>
            <!-- In case of search error unset token to hide table -->
            <unset token="displayTable"></unset>
          </error>
          <fail>
            <!-- In case of search failure unset token to hide table -->
            <unset token="displayTable"></unset>
          </fail>          
        </search>
      </table>
      <!-- Using rejects to show Custom Message when displayTable token is unset i.e. No Search Result, Search Error or Search Failure -->
      <html rejects="$displayTable$">
        <div style="font-size:150%;font-weight:bold;color:red;text-align:center">
          Search returned no results. Please consider changing search filters!
        </div>
      </html>
    </panel>
  </row>
</dashboard>

Refer to Splunk Documentation for Search Event Handlers, set and unset tokens,depends and rejects:
http://docs.splunk.com/Documentation/Splunk/latest/Viz/EventHandlerReference#Search_event_handlers
http://docs.splunk.com/Documentation/Splunk/latest/Viz/EventHandlerReference#unset

____________________________________________
| makeresults | eval message= "Happy Splunking!!!"
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...