Scheduled search strange behaviuor

[bckq]

This is my search:

index=cloud (cloud_severity="High" OR cloud_severity="Disaster") | dedup cloud_info,cloud_hostname | search ((cloud_status="PROBLEM" AND NOT cloud_hostname="*.c1a.net") OR (cloud_status="Error" AND cloud_hostname="admin@tattle")) | stats count as lista | rangemap field=lista low=0-0 default=severe

Time range is: from -30m@m to -1m@m
I have a dashboard with the result of this search presented in a table.
Search when there are no result should return 0, but I get:

http://i.imgur.com/9VL0355.png

When I press "View Result" I get this information:

http://i.imgur.com/nFKNEWq.png

The problem is that I get time range called "Before 6:45....". It should be like "Custom time" and on the bottom something like "1 result from 6:38:09 AM to 6:53:09 AM on Saturday".

I am using Splunk 5.0.6.

[bckq]

1) This is my search:

http://i.imgur.com/VJ9yxuk.png
As you can see, time range is ok. When I press Job Inspector I get fields for example:
searchEarliestTime 1391941200.000000000
searchLatestTime 1391942940.000000000
So it should work fine, but I don't know why there is written "before".

2) This is my dashboard:

<?xml version='1.0' encoding='utf-8'?>
<dashboard refresh="60">
  <label>Cloud Monitor</label>
  <row>
    <single>
      <searchName>CDP All errors and warnings 2</searchName>
      <option name="beforeLabel">CDP Total:</option>
      <option name="classField">range</option>
      <option name="count">10</option>
      <option name="displayRowNumbers">true</option>
      <option name="linkSearch"> | savedsearch "CDP ALL"</option>
      <option name="linkView">flashtimeline</option>
    </single>
    <table>
      <searchName>Cloud Monitor: All Disaster and High</searchName>
      <option name="beforeLabel">Cloud Total:</option>
      <option name="classField">range</option>
      <option name="count">10</option>
      <option name="displayRowNumbers">true</option>
      <option name="linkSearch">| savedsearch "Cloud: All Disaster and High"</option>
      <option name="linkView">flashtimeline</option>
    </table>
    <single>
      <searchName>Cloud Monitor Amazon: All Disaster and High</searchName>
      <option name="beforeLabel">Cloud Amazon:</option>
      <option name="classField">range</option>
      <option name="count">10</option>
      <option name="displayRowNumbers">true</option>
      <option name="linkSearch">| savedsearch "Cloud Amazon: All Disaster and High"</option>
      <option name="linkView">flashtimeline</option>
    </single>
  </row>
</dashboard>

I have problem only with second search. I changed it from Single Value to Table for debugging.

3) cloud_status could have state PROBLEM, ERROR or OK.

I get logs like:

cloud_hostname="host1.name.net" cloud__info="High IO disk usage" cloud_status="ERROR"

when problem disappear I get information:

cloud_hostname="host1.name.net" cloud__info="High IO disk usage" cloud_status="OK"

So when I search last 30 minutes and then dedup events I can check if there is ERROR or everything is OK. Then the last event is ERROR I get "stats count" higher than 0, when last cloud_status is OK the result should be 0.

I hope that I described it clearly.
What is also interesting, when I copy that problematic search and just paste it to the XML everything works good.

  <searchString>index=cloud (cloud_severity="High" OR cloud_severity="Disaster") | dedup cloud_info,cloud_hostname | search ((cloud_status="PROBLEM" AND NOT cloud_hostname="*.c1a.net") OR (cloud_status="Error" AND cloud_hostname="admin@tattle")) | stats count as lista | rangemap field=lista low=0-0 default=severe</searchString>
  <earliestTime>-30m@m</earliestTime>
  <latestTime>-1m@m</latestTime>

[sideview]

That is strange. I think you're having more than one problem here.

1) timerange.

If you look in Manager what timerange does Manager think the scheduled saved search has? Is it possible there's a typo on the earliest side? If the scheduled results are saying "before X", then that means it was run with no earliest time bound..

2) Can you post the XML of the dashboard?

Is it possible that there's a postprocess search hidden in there, or a forgotten dropdown that adds a searchterm via an intention or something? You are absolutely correct - no matter what happens before the | stats count as lista, the stats command will always output its single row. And rangemap will always take that row and add a range field, and give back one row. So the overall search result should always have one row, not zero rows... We're missing some piece of the puzzle here.

3) Separately, but also problematic, your dedup command is causing harm or at least not doing what you think it's doing.

| dedup cloud_info,cloud_hostname is going to go through the set of results it's given, and for each combination of cloud_info and cloud_hostname it sees, it is going to keep only the first such row it sees and throw away the rest. What values that row has for cloud_status is anybody's guess.. If for a given combination of cloud_info and cloud_hostname, the latest event happens to have cloud_status="Error", then it might match. If the "Error" rows are among the rows that get thrown away by dedup, you're outta luck. I would just remove that dedup, or possibly reduce it to dedup cloud_info cloud_hostname cloud_status

[sideview]

I'm sorry. The detail you added helps, but it looks like such an unusual problem that I would contact Splunk Support. Sounds like a bug.

As to the "before X" weirdness - since I wrote my answer I've seen that myself on scheduled searches. I think it represents some cosmetic bug in Splunk where scheduled search results no longer report their timerange properly to the Splunk UI.

But how it's possible for a search that essentially ends with a stats count, to ever return zero results - that one I have no idea about. Hopefully Splunk Support does. Good luck.

[bckq]

I added below answer to your points, because of lack of text formatting in comments.