Sort command causing lost records?

[buttsurfer]

I have a very simple search and when I add the sort command i lose almost 90% of my actual results. 

index="features" application=kokoapp type=userStats
| sort feature
| dedup feature
| table feature 

Without the sort command I get 35 results and with it included i only get 4 results. Is there something I am missing?

[gcusello]

Hi @buttsurfer,

sort cuts results only if you have more than 10,000 results.

probably is dedup that deletes some events.

Ciao.

Giuseppe

[buttsurfer]

Turns out there actually are over 10,000 results. But I need them all - is there a workaround to this?

[yuanliu]

You can try to raise limit in limits.conf.  But I don't understand what you mean by you need all >10,000 results because your dedup without sort only gives you 35.

If you want to sort by feature, sort after dedup. (Sort is memory-intensive; performing sort on any sizable chunk should always be deliberate.)

index="features" application=kokoapp type=userStats
| dedup feature
| sort feature
| table feature 

This should give you the same desired result.

[isoutamo]

Definitely you should 1st dedup and then sort. Otherwise there is possibility that you hit 10k limits and even that you wasted resources.

[PickleRick]

10k limit for sort is one thing but if you say that you get 4 out of 35 results, that's way under 10k 😉

Remember that dedup works by leaving first event with given field(s) and then dropping other subsequent results with this field(s). So it is most probably the culprit here. When you sort by a field then dedup by this field you'll effectively leave just one event per each possible value of this field.

[gcusello]

Hi @buttsurfer,

as you can read at https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Sort yu could use:

| sort 0 feature

in this way you don't limit results.

Ciao.

Giuseppe