topic Re: Help sorting by time results in lost records in Splunk Search

Help sorting by time results in lost records

echelon101 — Thu, 06 Sep 2018 18:54:11 GMT

When I do a sort, the records show up newest first. I will typically search for events on the duration of a week or a month. If I add "| sort time" or "| sort _time" , the records will show up oldest first. The count of events does not change but I am missing events from the first day or two.

For example, with the time picker selecting all of July

            (host="myfirewall") AND ("2018-07-01" OR "2018-07-02" OR "2018-07-03")

will return 89 records, including all 3 days.

However,

          (host="myfirewall") AND ("2018-07-01" OR "2018-07-02" OR "2018-07-03")  | SORT time

will return 89 records, oldest first, but does not include "2018-07-01"

Using
| SORT -time

will return 89 records, newest first, but does not include "2018-07-01"

Re: Help sorting by time results in lost records

horsefez — Thu, 06 Sep 2018 19:33:58 GMT

Always do | sort _time and tell me if after that the events still get lost in the void.

Re: Help sorting by time results in lost records

echelon101 — Tue, 29 Sep 2020 21:09:32 GMT

I have tried the following

| sort _time
| sort -_time
| sort time
| sort -time

with the same results.

If I look at an event log fields in a recent event I see that
time = 2018-08-27 08:14:26
_time = 2018-08-27T08:08:11.000-04:00

The problem may be occurring in search queries that are relatively complex (e.g. where I search for firewall events and have a log "NOT (this OR that OR ...) " statement to filter out events that aren't of interest. I tried to make sure that the entire query prior to the " | sort .... " entry was in ().

Re: Help sorting by time results in lost records

echelon101 — Fri, 07 Sep 2018 16:08:04 GMT

Thanks for the advice. This is really helping me get an idea of what I can do with splunk reporting

It seams whatever I do causes problems when I want to sort oldest first. I also found the "|reverse| command which is a little simpler since I don't have to worry I am messing up the date format string. I find that all the regular and reverse queries appear to yield the same number of records, when I export to CSV, the report lengths are quite different.

Adding "NOT" to my queries seems to be the culprit . For example , the following query is missing event records when I try to sort oldest first.

           (host=somehost AND "this_string"  NOT "that_string") | sort _time

But the following query is does show all records

           (host=somehost AND "this_string"  OR "that_string") | sort _time