How to avoid events not being returned in sub-second error?

[mahesh27]

Hi All,

getting following error in splunk:

"Events may not be returned in sub-second order due to search memory limits . See search.log for more information. settings: [search]:max_rawsize_perchunk"

when i am searching for paticular time range like : 4 to 8 i am getting this error.
but if i search for last 15 mins or 24 hours or last 7 days i am not getting the error.

I understood : that between 4 to 8 timerange there where lot events coming for one second.

1. below are my  props configured and sample logs:

20221012453012
20220812453012
20220912453012
20220612453012
H1S98765~~PR~;R ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
H1S98765~~PR~;Z ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
H1S98765~~PR~;M ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
H1S98765~~PR~;T ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV

[logs:health:app]
truncate=10000
time_prefix=(?:[^~]+~)~(?:[^~]+~){3}
time_format=%a %b %d %H: %M: %S  %Z
disable=false
max_timestamp_lookahead=75
charset=UFT_8
no_binary_check=true
datetime_config=CURRENT
should_linenerge=false
line_breaker=([\r\n]+)\w{8}~~
annotate_punct=false

2. below are my  props configured and sample logs:
[10/07/22 12:55:40"7451 IST] 89786545 medapplog  9[10/07/22 12:55:40"7451 IST-897654] [app=med, sucees=0, failed=10, validpoints=100]  the events are assocuiated with the med application user=app client=med
[08/07/22 12:55:40"7451 IST] 89786545 medapplog  9[10/07/22 12:55:40"7451 IST-897654] [app=med, sucees=0, failed=10, validpoints=100]  the events are assocuiated with the med application user=app client=med
[10/12/22 12:55:40"7451 IST] 89786545 medapplog  9[10/07/22 12:55:40"7451 IST-897654] [app=med, sucees=0, failed=10, validpoints=100]  the events are assocuiated with the med application user=app client=med

[logs:med:app]
time_prefix=^\[
time_format=%m %d %y  %H: %M: %S: %3Q  %Z
max_timestamp_lookahead=30
should_linenerge=false
line_breaker=([\r\n]+)\[\d{1,2}\/\d{1,2}\/\d{2}\s\d{1,2}:\d{2}:\d{2}:\d{3}\s\D{3}\]
truncate=99999

please let me know how to avoid this error coming when i search.

[richgalloway · Accepted solution]

This happens when more than 100MB of search results with the same timestamp are found.  How does that happen?  Either 1) all (or a lot) of your data has the same timestamp; or 2) it has no timestamp and Splunk assigns the same timestamp to it; or 3) the props.conf settings are incorrect, leading Splunk to assign the same wrong timestamp to all or a lot of the data.  In this case, the answer appears to be #3 (with some #2).

Of course, that could change based on what is in search.log.

Let's look at log #1.

Half of the sample events are just digits that might be a timestamp.  The other half contain a timestamp, but the props are a little off.  Try these (replacing only those mentioned below).  (FWIW, I always specify props.conf settings in upper case.)

TIME_PREFIX = (.*?~){5}
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y

Log #2 is similar.  The TIME_FORMAT setting does not match the sample data.  Try this one

TIME_FORMAT = %m/%d/%y %H:%M:%S"%4N %Z