Hi All,
getting following error in splunk:
"Events may not be returned in sub-second order due to search memory limits . See search.log for more information. settings: [search]:max_rawsize_perchunk"
when i am searching for paticular time range like : 4 to 8 i am getting this error.
but if i search for last 15 mins or 24 hours or last 7 days i am not getting the error.
I understood : that between 4 to 8 timerange there where lot events coming for one second.
1. below are my props configured and sample logs:
20221012453012
20220812453012
20220912453012
20220612453012
H1S98765~~PR~;R ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
H1S98765~~PR~;Z ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
H1S98765~~PR~;M ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
H1S98765~~PR~;T ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
[logs:health:app]
truncate=10000
time_prefix=(?:[^~]+~)~(?:[^~]+~){3}
time_format=%a %b %d %H: %M: %S %Z
disable=false
max_timestamp_lookahead=75
charset=UFT_8
no_binary_check=true
datetime_config=CURRENT
should_linenerge=false
line_breaker=([\r\n]+)\w{8}~~
annotate_punct=false
2. below are my props configured and sample logs:
[10/07/22 12:55:40"7451 IST] 89786545 medapplog 9[10/07/22 12:55:40"7451 IST-897654] [app=med, sucees=0, failed=10, validpoints=100] the events are assocuiated with the med application user=app client=med
[08/07/22 12:55:40"7451 IST] 89786545 medapplog 9[10/07/22 12:55:40"7451 IST-897654] [app=med, sucees=0, failed=10, validpoints=100] the events are assocuiated with the med application user=app client=med
[10/12/22 12:55:40"7451 IST] 89786545 medapplog 9[10/07/22 12:55:40"7451 IST-897654] [app=med, sucees=0, failed=10, validpoints=100] the events are assocuiated with the med application user=app client=med
[logs:med:app]
time_prefix=^\[
time_format=%m %d %y %H: %M: %S: %3Q %Z
max_timestamp_lookahead=30
should_linenerge=false
line_breaker=([\r\n]+)\[\d{1,2}\/\d{1,2}\/\d{2}\s\d{1,2}:\d{2}:\d{2}:\d{3}\s\D{3}\]
truncate=99999
please let me know how to avoid this error coming when i search.
This happens when more than 100MB of search results with the same timestamp are found. How does that happen? Either 1) all (or a lot) of your data has the same timestamp; or 2) it has no timestamp and Splunk assigns the same timestamp to it; or 3) the props.conf settings are incorrect, leading Splunk to assign the same wrong timestamp to all or a lot of the data. In this case, the answer appears to be #3 (with some #2).
Of course, that could change based on what is in search.log.
Let's look at log #1.
Half of the sample events are just digits that might be a timestamp. The other half contain a timestamp, but the props are a little off. Try these (replacing only those mentioned below). (FWIW, I always specify props.conf settings in upper case.)
TIME_PREFIX = (.*?~){5}
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y
Log #2 is similar. The TIME_FORMAT setting does not match the sample data. Try this one
TIME_FORMAT = %m/%d/%y %H:%M:%S"%4N %Z
This happens when more than 100MB of search results with the same timestamp are found. How does that happen? Either 1) all (or a lot) of your data has the same timestamp; or 2) it has no timestamp and Splunk assigns the same timestamp to it; or 3) the props.conf settings are incorrect, leading Splunk to assign the same wrong timestamp to all or a lot of the data. In this case, the answer appears to be #3 (with some #2).
Of course, that could change based on what is in search.log.
Let's look at log #1.
Half of the sample events are just digits that might be a timestamp. The other half contain a timestamp, but the props are a little off. Try these (replacing only those mentioned below). (FWIW, I always specify props.conf settings in upper case.)
TIME_PREFIX = (.*?~){5}
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y
Log #2 is similar. The TIME_FORMAT setting does not match the sample data. Try this one
TIME_FORMAT = %m/%d/%y %H:%M:%S"%4N %Z