rangemap with too many values

[tyronetv]

have this search:

application response sourcetype=log1 OR sourcetype=log2 (host = host1 OR host = host2 OR host = host3 OR host = host4 ) | rex "(A|a)pplication response.*time was\s+(?P<app_response>\d+)\s" | rangemap field=app_response "A. Less than 0.25 seconds"=0-249 "B. More than 0.25 but less than 0.5 seconds"=250-500 "C. More than half-second but less than a second"=500-1000 default="D. More than a second" |stats count by range

Should work, right? If I run it with just "stats count" I get 55,127 returns.

If I run it with rangemap I get 77,484 with 22,377 going to the "default" category.

If I do the search and and only search for items over 1000 ms I get zero ( "search app_response>1000").

So, why the extra bad numbers? What am I doing wrong?

[lguinn2 · Accepted solution]

Rangemap is a strange command - it is actually a custom command and written as a Python script. I would try this instead:

application response sourcetype=log1 OR sourcetype=log2 (host = host1 OR host = host2 OR host = host3 OR host = host4 ) 
| rex "(A|a)pplication response.*?time was\s+(?P<app_response>\d+)\s" 
| where app_response >= 0
| eval appResponseCategory = case(
          app_response<250,"A. Less than 0.25 seconds",
          app_response>=250 AND app_response<500,"B. More than 0.25 but less than 0.5 seconds"
          app_response>=500 AND app_response<1000,"C. More than half-second but less than a second"
          "1"=="1","D. One second or more" )

Note that I eliminated events that did not have an application response time - this may be where your "extra" default events were arising. Also, I made sure that the categories did not overlap, as your original categories did at 500 (one-half second). Finally, I think that the case function will out-perform the rangemap command.

[tyronetv]

This was the fix. I had used case before (another question) but was suggested I use rangemap. For this search using case is the clear winner.
Thanks.

[tyronetv]

2013-03-01 12:35:28,878 INFO [ler-HTTPThreadGroup-17042] RID=1362170128682-2299470 c.r.t.i.s.e.applicationImageArchiveConnection - application response time was 138 milliseconds.

2013-03-01 14:35:22,040[ndler-HTTPThreadGroup-681] INFO dis.service.application.ImageArchiveConnection - [1] application response time was 128 milliseconds.

2013-03-01 12:35:21,950 INFO [ler-HTTPThreadGroup-17053] RID=1362170121771-2299465 c.r.t.i.s.e.applicationImageArchiveConnection - application response time was 124 milliseconds.

[martin_mueller]

Can you provide anonymized sample data?

[tyronetv]

Maybe I wasn't clear. The default bucket has 20,000+ count that do not exist in the search. Read my initial post. Search pipe to count gives 50K+ results. Search pipe to rangemap gives 70K+ results.

[martin_mueller]

The default bucket contains all events that do not belong in another bucket. That's those >1000, those <0, and those with no value.

[tyronetv]

They have "no value" that I can find. There should be no counts in the default bucket. But, to more specifically answer, if you look at the query, I believe the default should include values > 1000

[martin_mueller]

What duration value(s) do those in the default bucket have?

[tyronetv]

No, the issue is that the 'default' bucket has 20,000+ that don't exist in the primary search string.

[martin_mueller]

Are there by any chance thousands with exactly 500ms duration? Those get rangemapped twice due to an overlap at 500.