Splunk Search

Sort command causing lost records?

buttsurfer
Path Finder

I have a very simple search and when I add the sort command i lose almost 90% of my actual results. 

 

 

index="features" application=kokoapp type=userStats
| sort feature
| dedup feature
| table feature 

 

 

 

Without the sort command I get 35 results and with it included i only get 4 results. Is there something I am missing?

Labels (1)
Tags (1)
0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @buttsurfer,

sort cuts results only if you have more than 10,000 results.

probably is dedup that deletes some events.

Ciao.

Giuseppe

0 Karma

buttsurfer
Path Finder

Turns out there actually are over 10,000 results. But I need them all - is there a workaround to this?

0 Karma

yuanliu
SplunkTrust
SplunkTrust

You can try to raise limit in limits.conf.  But I don't understand what you mean by you need all >10,000 results because your dedup without sort only gives you 35.

If you want to sort by feature, sort after dedup. (Sort is memory-intensive; performing sort on any sizable chunk should always be deliberate.)

index="features" application=kokoapp type=userStats
| dedup feature
| sort feature
| table feature 

This should give you the same desired result.

0 Karma

isoutamo
SplunkTrust
SplunkTrust

Definitely you should 1st dedup and then sort. Otherwise there is possibility that you hit 10k limits and even that you wasted resources.

0 Karma

PickleRick
SplunkTrust
SplunkTrust

10k limit for sort is one thing but if you say that you get 4 out of 35 results, that's way under 10k 😉

Remember that dedup works by leaving first event with given field(s) and then dropping other subsequent results with this field(s). So it is most probably the culprit here. When you sort by a field then dedup by this field you'll effectively leave just one event per each possible value of this field.

0 Karma

gcusello
SplunkTrust
SplunkTrust

Hi @buttsurfer,

as you can read at https://docs.splunk.com/Documentation/SplunkCloud/latest/SearchReference/Sort yu could use:

| sort 0 feature

in this way you don't limit results.

Ciao.

Giuseppe

0 Karma
Get Updates on the Splunk Community!

.conf24 | Day 0

Hello Splunk Community! My name is Chris, and I'm based in Canberra, Australia's capital, and I travelled for ...

Enhance Security Visibility with Splunk Enterprise Security 7.1 through Threat ...

(view in My Videos)Struggling with alert fatigue, lack of context, and prioritization around security ...

Troubleshooting the OpenTelemetry Collector

  In this tech talk, you’ll learn how to troubleshoot the OpenTelemetry collector - from checking the ...