Splunk Search

Errors through REST API but not through Splunk GUI

mlevsh
Builder

One of our teams is running  Java script that uses REST API to fetch data from Splunk Cloud using the search.
They run it around 1am EST, once per day. Usually it runs successfully but sometimes, not clear due to what reason, the script fails due to the following error:

"Events might not be returned in sub-second order due to search memory limits. See search.log for more information. Increase the value of the following limits.conf setting:[search]:max_rawsize_perchunk."

Here is the more detailed output from the script log:

2020-09-09 01:59:17,172 [INFO ] [mcsplunk] Connecting to splunk
    2020-09-09 01:59:17,173 [INFO ] [mcsplunk] Setting up proxy to establish connection
    2020-09-09 01:59:17,234 [INFO ] [mcsplunk] Successfully established connection
with Splunk on host "our_host" using User "our_user" in app conext "our_app_name"
    2020-09-09 01:59:17,234 [INFO ] [mcsplunk] Starting to fetch data from Splunk
    2020-09-09 01:59:29,740 [INFO ] [mcsplunk] Splunk query execution has completed.
Scanned:723492, Matched:723036,Results:1, Run Duration:11.146
    09-09-2020 05:01:28.899 +0000 ERROR SearchMessages - orig_component="CursoredSearch"
sid="1234567.123jobid"     peer_name="indexer01.ourdomain.com" message=[indexer01.ourdomain.com] Events might not be returned in sub-second order due to search memory limits. See search.log for more information. Increase the value of the following limits.conf setting:[search]:max_rawsize_perchunk.

But when we attempt to run the same splunk search in Splunk Cloud GUI - it works fine, no errors. 

Any idea what could be the cause of such behavior? 

Labels (1)
0 Karma

mlevsh
Builder

@thambisetty  thank you for suggestion. The disk quota was already 5000Mb, we increased it to 7500MB and it didn't help. Still getting the same error. Script fails as a result

When we login to Splunk Cloud GUI with the user id that the scheduled script runs with - it works without giving us the error. 

0 Karma

thambisetty
SplunkTrust
SplunkTrust

did you try same search with same time range for which you are seeing error from script?

 

————————————
If this helps, give a like below.
Tags (2)
0 Karma

thambisetty
SplunkTrust
SplunkTrust

look into below thread.

https://community.splunk.com/t5/Installation/quot-Events-may-not-be-returned-in-sub-second-order-due...

————————————
If this helps, give a like below.
0 Karma

thambisetty
SplunkTrust
SplunkTrust

Search is returning more results which is actually more than user's disk space quota set in Role of that particular user used for authentication in rest query.

to increase user quota ( disk space limit) , open roles under settings and choose the role of the user and change quota. 

I would also recommend to follow best practices to improve search performance and have more work load on indexers rather on search head ( I believe you search is bringing all results to search head and then results are processed).

————————————
If this helps, give a like below.
0 Karma
Get Updates on the Splunk Community!

Unlock New Opportunities with Splunk Education: Explore Our Latest Courses!

At Splunk Education, we’re dedicated to providing top-tier learning experiences that cater to every skill ...

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...