topic Events might not be returned in sub-second order due to search memory limits in Installation

Events might not be returned in sub-second order due to search memory limits

rayar — Sun, 03 Jan 2021 10:30:57 GMT

we are facing the below issue for searches

please advise what I can reconfigure to avoid this errors ?

4 errors occurred while the search was executing. Therefore, search results might be incomplete.

[ilissplidx01] Events might not be returned in sub-second order due to search memory limits. See search.log for more information. Increase the value of the following limits.conf setting:[search]:max_rawsize_perchunk.
[ilissplidx02] Events might not be returned in sub-second order due to search memory limits. See search.log for more information. Increase the value of the following limits.conf setting:[search]:max_rawsize_perchunk.
[ilissplidx06] Events might not be returned in sub-second order due to search memory limits. See search.log for more information. Increase the value of the following limits.conf setting:[search]:max_rawsize_perchunk.

[splunk@ilisspldepl01 deployment-apps]$ cat ./AM_all_indexers_tuning/local/limits.conf
[default]
max_mem_usage_mb = 600
#
[search]
#dispatch_dir_warning_size = 3500
base_max_searches = 60
# # ERROR: Events may not be returned in sub-second order due to memory pressure.
max_rawsize_perchunk = 200000000
#
[pdf]
max_rows_per_table = 10000
#
[scheduler]
max_searches_perc = 100
#
[join]
subsearch_maxout = 500000
#
[realtime]
indexed_realtime_use_by_default = true
[splunk@ilisspldepl01 deployment-apps]$

[ilissplidx08] Events might not be returned in sub-second order due to search memory limits. See search.log for more information. Increase the value of the following limits.conf setting:[search]:max_rawsize_perchunk.

[splunk@ilissplidx01 ~]$ grep MemTotal /proc/meminfo
MemTotal: 65688816 kB
[splunk@ilissplidx01 ~]$
[rayar@ilissplidx08 ~]$ grep MemTotal /proc/meminfo
MemTotal: 528052452 kB
[rayar@ilissplidx08 ~]$

Re: Events might not be returned in sub-second order due to search memory limits

scelikok — Sun, 03 Jan 2021 12:23:42 GMT

Hi @rayar,

Actually this is a warning that you probably face when you run a search that gets all raw data. If you run a search that has statistics command like stats, you would not see that error.

I suggest to check event ingestion if timestamps are correctly parsed. This may due to large numbers of events with the exact same timestamp -- possibly caused by non-timestamped events that are being timestamp by Splunk as they are indexed.

If timestamps are ok, you want to retrieve all raw data on search you can try increasing "max_rawsize_perchunk" to much higher value.

If this reply helps you an upvote is appreciated.

Re: Events might not be returned in sub-second order due to search memory limits

rayar — Sun, 03 Jan 2021 13:45:17 GMT

this index contains the data with current_time sourcetype

the question what value you would recommend to set for max_rawsize_perchunk

Re: Events might not be returned in sub-second order due to search memory limits

scelikok — Mon, 04 Jan 2021 08:56:50 GMT

It depends on your events size and number of events per seconds. You can try by 200000000 increments.

Re: Events might not be returned in sub-second order due to search memory limits

microgag — Wed, 30 Jun 2021 05:14:56 GMT

There are similar post on the following issue:

https://community.splunk.com/t5/Installation/quot-Events-may-not-be-returned-in-sub-second-order-due-to/m-p/136708