How to finetune subsearch

[NAGA4]

I have below requirement. I am working on two types of events. 

Source 1 - From here I wanted to take employee email and url. Then using lookup I am taking emp id for respective email.  And from url am fetching domain. These are the two key fields (empid and domain) i need to compare with Source 2. Along with this I need to take other set of fields too like sender, clickip and some others

Source 2 - From here i just need to take status of particular event - Status can be allow or block

index=source2 userid=$empid$ domain=$domain$ | stats values(status) 

I tried below
1. I used map and taking results from source 1 and passing it via map command to source. then did stats. It works fine. But problem is am unable to enable the durablitity for this saved search. is there any way to do this?

2. So I used append. I used both searches with append and taking the stats. Here i evaluated a new field called type in both sources and kept type=source1 and type=source2 in both queries and then took status values(type) and mvcount(type) = 2 then am taking results. Thsi is too working fine but hit the subsearch count 50k in the limit. I am unable to increase this limit as of now due to restricitions . Is there any other workaround

[yuanliu]

As you will find everywhere in this forum, map is usually not the solution; subsearch also should not be the default go to.  What you ought to ask yourself are:

• What is my input?
• What is my desired output?
• What is the logical relationship between input and desired output?

Once you can verbalize these, you will find that stats is often all you need.  But in your case, you are not interested in source1 at all.  So, subsearch could be of good use - but not with map or append.

Let me see if I get your question correct.  You have two sources, source1 has two fields of interest, empid, domain; source2 has three, userid, domain, and status.  All you want is a list of unique values of status - like 200, 203, 400, 404, 500, 501, as long as they are from index=source2 that matches any and all allowable combinations of empid and domain from index=source1 provided that userid in source2 equaling an allowable empid in source1.

If the above is correct, the simplest approach would be

index = source2
| tojson output_field=user_domain userid domain
| search
  [search index = source1
    | fields empid domain
    | rename empid as userid
    | stats count by userid domain ``` can be simplified to dedup but performance may suffer ```
    | fields - count
    | tojson output_field=user_domain userid domain]
| stats values(status)

Here, subsearch is used as a filter, not going through append.  So, the 50K limit doesn't apply.

 A more traditional approach with no subsearch could be

index IN (source1, source2)
| eval userid = coalesce(userid, empid)
| stats values(status) as status values(index) as sources by userid domain
| where sources = "source1" ``` make sure to only count empid-domain combo that do appear in source1 ```
| stats values(status)

Hope this helps.

[ITWhisperer]

This is a little tricky to follow exactly what you are trying to do. Please provide some obfuscated / desensitised sample events from both sources in raw and unformatted mode, preferably in a code block </> so they can be copied and pasted into a makeresults command to enable us to try out some potential solutions for you. It is nearly always the most expedient way to approach requesting help with searches.

Having said that, you could try including both sources on the initial search line which would avoid the subsearch limitations, you just then need to be able to filter and combine the events (hence the need to see what you are dealing with!).