topic Re: Inconsistent Search results in Splunk Search

Inconsistent Search results

Kate_Lawrence-G — Tue, 28 Feb 2012 17:00:41 GMT

Hi,

I am having some inconsistent search results and I'm not terribly sure why.

search #1:

earliest=-7d latest=-2h sourcetype=x  type=delta status=fatal | ctable type status

Which returns:

    type    fatal   TOTAL
1   delta   658     658
2   TOTAL   658     658

search #2:

earliest=-7d latest=-2h sourcetype=x  type=* status=fatal | ctable type status

Which returns:

    type       fatal    TOTAL
1   delta      861      861
2   full       34       34
3   TOTAL      895      895

The searches are running over the same period of time but returning differing results. I've confirmed that all my indexers are available.

Thanks

Kate

Re: Inconsistent Search results

Stephen_Sorkin — Tue, 28 Feb 2012 18:35:58 GMT

what results do you get if you change your first search to: earliest=-7d latest=-2h sourcetype=x status=fatal | search type=delta | ctable type status

Re: Inconsistent Search results

Kate_Lawrence-G — Tue, 28 Feb 2012 21:00:07 GMT

I get the 861 count again....

     type   fatal    TOTAL

1 delta 861 861
2 TOTAL 861 861

Re: Inconsistent Search results

Stephen_Sorkin — Mon, 05 Mar 2012 17:29:47 GMT

When you search for type=delta in part of the search before the first pipe character, we use the heuristic optimization that the value of the field/value comparison, here delta is indexed. I presume that this heuristic fails here. You can fix this by changing "type" in fields.conf to not be an indexed field, at the consequence of some searches being slower.

You can find the problematic events using:

earliest=-7d latest=-2h sourcetype=x  NOT delta status=fatal | search type=delta