Monitoring Splunk

Hitting eventstats memory limit. Is there another way to form my search?

jedatt01
Builder

I'm seeing the following error below when trying to use eventstats on a large lookup table (about 500,000 rows).

06-24-2016 15:44:04.735 ERROR StatsProcessor - Reached limit max_mem_usage_mb (200 MB), results may be incomplete! Please increase the max_mem_usage_mb in limits.conf .

Getting the max_mem_usage_mb increase will take a long time as I'm a user in a large enterprise environment that would require testing first.

In the lookup table, I have records of machine check-ins to an DM system. I'm trying to create a search that will look for the latest check-in in the lookup and remove the older check-ins. My current search below is not completing because of the memory limit.

| inputlookup mdm_master | eventstats max(MDM_Last_Check_in_epoc) as last_check by MDM_MAC_Address MDM_Server | where MDM_Last_Check_in_epoc = last_check | table MDM* | outputlookup mdm_master

Can anybody think of a more efficient way to do this that will take less memory?

0 Karma

maciep
Champion

Not sure how these things work under the hood, so this may be as or more inefficient, but you could try using sort/dedup.

| inputlookup mdm_master 
| eval mdm_mac_server = MDM_MAC_Address." ".MDM_Server
| sort mdm_mac_server - MDM_Last_Check_in_epoc
| dedup mdm_mac_server
| table MDM*
| outputlookup mdm_master

Also, again not sure how efficient, but you could try to use a stats instead of eventstats, and then lookup or join the results to the lookup again, keep the ones that match output the lookup again?

0 Karma

jedatt01
Builder

must be hardcoded somewhere. I tried sort 0 and also sort 500000 and it still tells me it's being limited to 10k results.

0 Karma

maciep
Champion

interesting...maybe it's in limits somewhere.

What about looking up from the lookup? Something like this?

| inputlookup mdm_master 
| tstats max(MDM_Last_Check_in_epoc) as MDM_Last_Check_in_epoc
| lookup mdm_master MDM_Last_Check_in_epoc
| table MDM* 
| outputlookup mdm_master
0 Karma

jedatt01
Builder

Unfortunately, sort has a 10,000 event limit. No good there either.

0 Karma

maciep
Champion

Oh, whoops....you can use sort 0 to remove the default limit, like

| sort 0 mdm_mac_server - MDM_Last_Check_in_epoc
0 Karma

aljohnson_splun
Splunk Employee
Splunk Employee

I sort of wonder if you could use streamstats and just get the last calculation - but to be honest I don't really know. Could be worth a try ?

0 Karma

woodcock
Esteemed Legend

2 options, both are higher-level things:

1: Setup a Data Model and accelerate it then convert your search to tstats.
2: Setup a Summary Index.

0 Karma
Get Updates on the Splunk Community!

Synthetic Monitoring: Not your Grandma’s Polyester! Tech Talk: DevOps Edition

Register today and join TekStream on Tuesday, February 28 at 11am PT/2pm ET for a demonstration of Splunk ...

Instrumenting Java Websocket Messaging

Instrumenting Java Websocket MessagingThis article is a code-based discussion of passing OpenTelemetry trace ...

Announcing General Availability of Splunk Incident Intelligence!

Digital transformation is real! Across industries, companies big and small are going through rapid digital ...