Splunk Enterprise

How to avoid events not being returned in sub-second error?

mahesh27
Communicator

Hi All,

getting following error in splunk:

"Events may not be returned in sub-second order due to search memory limits . See search.log for more information. settings: [search]:max_rawsize_perchunk"

when i am searching for paticular time range like : 4 to 8 i am getting this error.
but if i search for last 15 mins or 24 hours or last 7 days i am not getting the error.

I understood : that between 4 to 8 timerange there where lot events coming for one second.

1. below are my  props configured and sample logs:

20221012453012
20220812453012
20220912453012
20220612453012
H1S98765~~PR~;R ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
H1S98765~~PR~;Z ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
H1S98765~~PR~;M ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV
H1S98765~~PR~;T ESC~AB~Thu Oct 12 12:34:56 IST 2022~B~1.22~2.22~3456.98~GF~4356BV

[logs:health:app]
truncate=10000
time_prefix=(?:[^~]+~)~(?:[^~]+~){3}
time_format=%a %b %d %H: %M: %S  %Z
disable=false
max_timestamp_lookahead=75
charset=UFT_8
no_binary_check=true
datetime_config=CURRENT
should_linenerge=false
line_breaker=([\r\n]+)\w{8}~~
annotate_punct=false

 

2. below are my  props configured and sample logs:
[10/07/22 12:55:40"7451 IST] 89786545 medapplog  9[10/07/22 12:55:40"7451 IST-897654] [app=med, sucees=0, failed=10, validpoints=100]  the events are assocuiated with the med application user=app client=med
[08/07/22 12:55:40"7451 IST] 89786545 medapplog  9[10/07/22 12:55:40"7451 IST-897654] [app=med, sucees=0, failed=10, validpoints=100]  the events are assocuiated with the med application user=app client=med
[10/12/22 12:55:40"7451 IST] 89786545 medapplog  9[10/07/22 12:55:40"7451 IST-897654] [app=med, sucees=0, failed=10, validpoints=100]  the events are assocuiated with the med application user=app client=med

[logs:med:app]
time_prefix=^\[
time_format=%m %d %y  %H: %M: %S: %3Q  %Z
max_timestamp_lookahead=30
should_linenerge=false
line_breaker=([\r\n]+)\[\d{1,2}\/\d{1,2}\/\d{2}\s\d{1,2}:\d{2}:\d{2}:\d{3}\s\D{3}\]
truncate=99999


please let me know how to avoid this error coming when i search.



Labels (1)
0 Karma
1 Solution

richgalloway
SplunkTrust
SplunkTrust

This happens when more than 100MB of search results with the same timestamp are found.  How does that happen?  Either 1) all (or a lot) of your data has the same timestamp; or 2) it has no timestamp and Splunk assigns the same timestamp to it; or 3) the props.conf settings are incorrect, leading Splunk to assign the same wrong timestamp to all or a lot of the data.  In this case, the answer appears to be #3 (with some #2).

Of course, that could change based on what is in search.log.

Let's look at log #1.

Half of the sample events are just digits that might be a timestamp.  The other half contain a timestamp, but the props are a little off.  Try these (replacing only those mentioned below).  (FWIW, I always specify props.conf settings in upper case.)

TIME_PREFIX = (.*?~){5}
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y

Log #2 is similar.  The TIME_FORMAT setting does not match the sample data.  Try this one

TIME_FORMAT = %m/%d/%y %H:%M:%S"%4N %Z

 

 

 

---
If this reply helps you, Karma would be appreciated.

View solution in original post

richgalloway
SplunkTrust
SplunkTrust

This happens when more than 100MB of search results with the same timestamp are found.  How does that happen?  Either 1) all (or a lot) of your data has the same timestamp; or 2) it has no timestamp and Splunk assigns the same timestamp to it; or 3) the props.conf settings are incorrect, leading Splunk to assign the same wrong timestamp to all or a lot of the data.  In this case, the answer appears to be #3 (with some #2).

Of course, that could change based on what is in search.log.

Let's look at log #1.

Half of the sample events are just digits that might be a timestamp.  The other half contain a timestamp, but the props are a little off.  Try these (replacing only those mentioned below).  (FWIW, I always specify props.conf settings in upper case.)

TIME_PREFIX = (.*?~){5}
TIME_FORMAT = %a %b %d %H:%M:%S %Z %Y

Log #2 is similar.  The TIME_FORMAT setting does not match the sample data.  Try this one

TIME_FORMAT = %m/%d/%y %H:%M:%S"%4N %Z

 

 

 

---
If this reply helps you, Karma would be appreciated.
Get Updates on the Splunk Community!

Developer Spotlight with William Searle

The Splunk Guy: A Developer’s Path from Web to Cloud William is a Splunk Professional Services Consultant with ...

Major Splunk Upgrade – Prepare your Environment for Splunk 10 Now!

Attention App Developers: Test Your Apps with the Splunk 10.0 Beta and Ensure Compatibility Before the ...

Stay Connected: Your Guide to June Tech Talks, Office Hours, and Webinars!

What are Community Office Hours?Community Office Hours is an interactive 60-minute Zoom series where ...