Splunk Search

subsearch results truncated as the number of entries in the lookup exceeded 10000

sravipati
New Member

Hi all,

Using Splunk cloud I'm trying to look up the time difference between when a message is received from a sender and was delivered to a recipient. I have a lookup which has all the message ids totaling about 400k entries. For each message id, I'm interested in finding how long it took for the server to process the message.  Splunk seems to truncate the subsearch to 10000.  Here is what was noticed for the search below:

Subsearch produced 423340 results, truncating to maxout 10000.


index="stage" (host="msgsrv*" source="/var/log/messaging/msg.log" [|inputlookup sept-messages | fields id ]
(

( event=msg_rcvd AND "tag=""body""") OR ( event=msg_sent AND "tag=""body""") OR (event=msg_sent AND "tag=""result""")
)
)

|reverse
|eval msg_id = id
|eval msg_rcvd_time=if(event == "msg_rcvd", _time, 999999999999.999)
|eval client_out_time=if(event == "msg_sent", _time,999999999999.999)
|stats values(msg_rcvd_time) AS ins values(msg_sent_time) AS outs values(_time) AS times values(_raw) as raws values(to_user) AS to_users values(from_user) AS from_users by msg_id, to_user
|eval first_msg_rcvd_time = mvindex(mvsort(ins), 0)
|eval first_msg_sent_time = mvindex(mvsort(outs), 0)
|eval delta = first_msg_sent_time - first_msg_rcvd_time

How could the large lookup be processed and any suggestions for improving the above query?

Labels (4)
0 Karma

inventsekar
SplunkTrust
SplunkTrust

There should be some way about picking up only the first and last event and finding the time difference.. i am not sure how to do that. 

meanwhile, if you want to update the subsearch limitation of 10000 (splunk cloud customers will need to contact splunk support)

limits.conf file:

[subsearch]
* This stanza controls subsearch results.
* NOTE: This stanza DOES NOT control subsearch results when a subsearch is called by
commands such as join, append, or appendcols.
* Read more about subsearches in the online documentation:
http://docs.splunk.com/Documentation/Splunk/latest/Search/Aboutsubsearches

maxout =
* Maximum number of results to return from a subsearch.
* This value cannot be greater than or equal to 10500.
* Defaults to 10000.

thanks and best regards,
Sekar

PS - If this or any post helped you in any way, pls consider upvoting, thanks for reading !
0 Karma

sravipati
New Member

Thank you. Sure will look into that option of increasing. Wondering how large of a value is acceptable though? 

0 Karma
Get Updates on the Splunk Community!

Technical Workshop Series: Splunk Data Management and SPL2 | Register here!

Hey, Splunk Community! Ready to take your data management skills to the next level? Join us for a 3-part ...

Spotting Financial Fraud in the Haystack: A Guide to Behavioral Analytics with Splunk

In today's digital financial ecosystem, security teams face an unprecedented challenge. The sheer volume of ...

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability

Solve Problems Faster with New, Smarter AI and Integrations in Splunk Observability As businesses scale ...