Monitoring Splunk

Hitting eventstats memory limit. Is there another way to form my search?

Builder

I'm seeing the following error below when trying to use eventstats on a large lookup table (about 500,000 rows).

06-24-2016 15:44:04.735 ERROR StatsProcessor - Reached limit max_mem_usage_mb (200 MB), results may be incomplete! Please increase the max_mem_usage_mb in limits.conf .

Getting the max_mem_usage_mb increase will take a long time as I'm a user in a large enterprise environment that would require testing first.

In the lookup table, I have records of machine check-ins to an DM system. I'm trying to create a search that will look for the latest check-in in the lookup and remove the older check-ins. My current search below is not completing because of the memory limit.

| inputlookup mdm_master | eventstats max(MDM_Last_Check_in_epoc) as last_check by MDM_MAC_Address MDM_Server | where MDM_Last_Check_in_epoc = last_check | table MDM* | outputlookup mdm_master

Can anybody think of a more efficient way to do this that will take less memory?

0 Karma

Champion

Not sure how these things work under the hood, so this may be as or more inefficient, but you could try using sort/dedup.

| inputlookup mdm_master 
| eval mdm_mac_server = MDM_MAC_Address." ".MDM_Server
| sort mdm_mac_server - MDM_Last_Check_in_epoc
| dedup mdm_mac_server
| table MDM*
| outputlookup mdm_master

Also, again not sure how efficient, but you could try to use a stats instead of eventstats, and then lookup or join the results to the lookup again, keep the ones that match output the lookup again?

0 Karma

Builder

must be hardcoded somewhere. I tried sort 0 and also sort 500000 and it still tells me it's being limited to 10k results.

0 Karma

Champion

interesting...maybe it's in limits somewhere.

What about looking up from the lookup? Something like this?

| inputlookup mdm_master 
| tstats max(MDM_Last_Check_in_epoc) as MDM_Last_Check_in_epoc
| lookup mdm_master MDM_Last_Check_in_epoc
| table MDM* 
| outputlookup mdm_master
0 Karma

Builder

Unfortunately, sort has a 10,000 event limit. No good there either.

0 Karma

Champion

Oh, whoops....you can use sort 0 to remove the default limit, like

| sort 0 mdm_mac_server - MDM_Last_Check_in_epoc
0 Karma

Splunk Employee
Splunk Employee

I sort of wonder if you could use streamstats and just get the last calculation - but to be honest I don't really know. Could be worth a try ?

0 Karma

Esteemed Legend

2 options, both are higher-level things:

1: Setup a Data Model and accelerate it then convert your search to tstats.
2: Setup a Summary Index.

0 Karma
Don’t Miss Global Splunk
User Groups Week!

Free LIVE events worldwide 2/8-2/12
Connect, learn, and collect rad prizes
and swag!