Join truncating from too many results.

[dennywebb]

i have 2 kinds of logs, one for an install of a toolbar, and one for the USE of the toolbar to do a search.

the install log has the browser type, and the search log does not. both have a UniqueBrowserID field that will match them for a join.

my goal is to view the search log with the browser type. i have tried doing:

eventtype="SearchLog" sourcetype="apache_error" | join UniqueBrowserID [search eventtype="InstallLog" | fields BrowserName]

but always get this message:
[subsearch]: Subsearch produced 50000 results, truncating to maxout 50000.

and have results missing the BrowserName....

i tried making a macro and using an eval, but I can't figure out how to make a search in the macro only return a string so i get:
Error in 'SearchParser': The definition of macro 'BrowserNameByUniqueID(1)' is expected to be an eval expression that returns a string.

the macro is defined as:

eventtype="InstallLog" UniqueBrowserID="$UBI$" | head 1 | table BrowserName

Help!

[sideview]

I'm pretty sure you neither want nor need a join here.

(sourcetype="apache_error" eventtype="SearchLog" ) OR (eventtype="InstallLog") 
| stats values(BrowserName) count by UniqueBrowserID

If the two logs extract the browserId field with different fieldnames, you may need a little eval to normalize them. eg:

(sourcetype="apache_error" eventtype="SearchLog" ) OR (eventtype="InstallLog") 
| eval eval normalizedId=if(sourcetype="apache_error",browserID,uniq_browser_id)
| stats values(BrowserName) count by normalizedId

[marcoscala]

You can override the 50.000 default maximum number of events returned from the join subsearch in limits.conf using

[join]
subsearch_maxout = <yourvalue>

Marco

[MuS]

Just a small note: this question was tagged with splunkstorm. If you're on Splunk Storm this will not be possible, while for Splunk Enterprise this correct 😉

[lguinn2]

Try this

eventtype="SearchLog" sourcetype="apache_error" 
| join UniqueBrowserID 
         [ search eventtype="InstallLog" BrowserName=* 
         | addinfo | where _time >= info_min_time AND _time <= info_max_time
         | dedup UniqueBrowserID 
         | fields UniqueBrowserID BrowserName ]

Which may solve both the subsearch limit and the fact that you have blank BrowserNames. I am not sure why you need a macro - are you trying to run this search for a single browser ID?

Note that the inner search runs over all time by default. The search above uses the addinfo command to retrieve the min and max times from the outer search and applies them to the inner search in the where command.

[dennywebb]

my goal here is that we also have a bunch of searches coming from a web page, the logs of these DO include the browser with every log. i'd like to create a final search that includes BOTH types of logs and gives stats/charts/etc, including on browser types, for my dashboard.

[dennywebb]

and it's eliminating records if i do that... because an install may have happened a year ago that i'm trying to join with a search log to get the browser name. the above limits is to only installs in the same timeframe as the search.

[dennywebb]

i was trying to use a macro as a subsearch to get around the subsearch limitations... i though if i could do an eval macro for each record, then i WOULD only be looking for a single ID each time the eval ran (as i understand, that evals per record)... making it a smaller list.

i'm still being truncated if i go over "last 24 hours" with your solution.