I'm seeing the following error below when trying to use eventstats on a large lookup table (about 500,000 rows).
06-24-2016 15:44:04.735 ERROR StatsProcessor - Reached limit max_mem_usage_mb (200 MB), results may be incomplete! Please increase the max_mem_usage_mb in limits.conf .
Getting the max_mem_usage_mb increase will take a long time as I'm a user in a large enterprise environment that would require testing first.
In the lookup table, I have records of machine check-ins to an DM system. I'm trying to create a search that will look for the latest check-in in the lookup and remove the older check-ins. My current search below is not completing because of the memory limit.
| inputlookup mdm_master | eventstats max(MDM_Last_Check_in_epoc) as last_check by MDM_MAC_Address MDM_Server | where MDM_Last_Check_in_epoc = last_check | table MDM* | outputlookup mdm_master
Can anybody think of a more efficient way to do this that will take less memory?
Not sure how these things work under the hood, so this may be as or more inefficient, but you could try using sort/dedup.
| inputlookup mdm_master | eval mdm_mac_server = MDM_MAC_Address." ".MDM_Server | sort mdm_mac_server - MDM_Last_Check_in_epoc | dedup mdm_mac_server | table MDM* | outputlookup mdm_master
Also, again not sure how efficient, but you could try to use a stats instead of eventstats, and then lookup or join the results to the lookup again, keep the ones that match output the lookup again?
interesting...maybe it's in limits somewhere.
What about looking up from the lookup? Something like this?
| inputlookup mdm_master | tstats max(MDM_Last_Check_in_epoc) as MDM_Last_Check_in_epoc | lookup mdm_master MDM_Last_Check_in_epoc | table MDM* | outputlookup mdm_master