topic Re: Sort command causing lost records in Splunk Search

Sort command causing lost records?

buttsurfer — Mon, 20 Mar 2023 14:43:31 GMT

I have a very simple search and when I add the sort command i lose almost 90% of my actual results.

index="features" application=kokoapp type=userStats | sort feature | dedup feature | table feature

Without the sort command I get 35 results and with it included i only get 4 results. Is there something I am missing?

Re: Sort command causing lost records

gcusello — Fri, 17 Mar 2023 16:11:36 GMT

Hi @buttsurfer,

sort cuts results only if you have more than 10,000 results.

probably is dedup that deletes some events.

Ciao.

Giuseppe

Re: Sort command causing lost records

buttsurfer — Fri, 17 Mar 2023 16:14:38 GMT

Turns out there actually are over 10,000 results. But I need them all - is there a workaround to this?

Re: Sort command causing lost records

gcusello — Fri, 17 Mar 2023 16:30:32 GMT

Hi @buttsurfer,

as you can read at https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Sort yu could use:

| sort 0 feature

in this way you don't limit results.

Ciao.

Giuseppe

Re: Sort command causing lost records

PickleRick — Fri, 17 Mar 2023 16:57:38 GMT

10k limit for sort is one thing but if you say that you get 4 out of 35 results, that's way under 10k 😉

Remember that dedup works by leaving first event with given field(s) and then dropping other subsequent results with this field(s). So it is most probably the culprit here. When you sort by a field then dedup by this field you'll effectively leave just one event per each possible value of this field.

Re: Sort command causing lost records

yuanliu — Fri, 17 Mar 2023 23:35:45 GMT

You can try to raise limit in limits.conf. But I don't understand what you mean by you need all >10,000 results because your dedup without sort only gives you 35.

If you want to sort by feature, sort after dedup. (Sort is memory-intensive; performing sort on any sizable chunk should always be deliberate.)

index="features" application=kokoapp type=userStats | dedup feature | sort feature | table feature

This should give you the same desired result.

Re: Sort command causing lost records

isoutamo — Fri, 17 Mar 2023 23:40:38 GMT

Definitely you should 1st dedup and then sort. Otherwise there is possibility that you hit 10k limits and even that you wasted resources.