How do I get around the issue of the Segmentation and Subsearch limit if I have 30000 results?

[DanielFordWA]

Hi,

I have a index of raw usage data (iis) and a separate index of entitlement data (rest_ent_prod), both indexes have a unique identifier for each user "GUID".

I would like to be able to ad hoc search the raw usage index for user behavior of users with certain entitlements and also create summary indexes.

The issue I have is the number of unique users with certain entitlements is around 30k and subsearches max out at 10.5k.

Can anyone advise how I can get around this issue?

Thanks,

Dan

[woodcock]

The answer by @jeffland is absolutely the correct way but if you cannot make that work, and you can deal with using a 2-stage process to pump some of your data out to file or KV-store (using outputlookup), then you can use this trick to escape append/subsearch limits:

https://answers.splunk.com/answers/318428/how-can-i-escape-the-50k-subsearch-limit-while-lin.html

[jeffland · Accepted solution]

Joining may be more comfortable, but you can always get the same mechanics going with a simple stats on a search comprising both sources, split by the field you would usually join on. I.e., instead of

index=iis | join GUID [search index=rest_ent_prod]

you would do

index=iis OR index=rest_ent_prod | stats values(something) by GUID

Check this cool post for more detail!