Monitoring Splunk
Provide Splunk Cloud feedback in this confidential UX survey by June 17
for a chance to win a $200 Amazon gift card!

Mvexpand memory usage

333_gloom_333
Engager

Hello, everyone!

I was encountered with weird problem. I have the following search:

| tstats `summariesonly` count by source, host, index, sourcetype | table source, host, index, sourcetype | stats values(host) as src_host dc(host) as count by source, index | sort + count 
| mvexpand src_host
| outputlookup sourcescheck

Long story short, i got the error:

warn : command.mvexpand: output will be truncated at 5200 results due to excessive memory usage. Memory threshold of 500MB as configured in limits.conf / [mvexpand] / max_mem_usage_mb has been reached.

and my truncated sourcescheck.csv, which size was less then 1Mb (sic!)

Ok, we can see in docs (https://docs.splunk.com/Documentation/Splunk/7.3.1/SearchReference/Mvexpand):

The total necessary memory is the average result size multiplied by the number of results in the chunk multiplied by the average size of the multivalue field being expanded.

My question: is it the practical reality that for processing less that 1Mb of data with mvexpand command we need more than half of 1Gb of memory? Are there some workarounds or optimized approaches?

Thank you for any answers!

Labels (1)
0 Karma
1 Solution

kamlesh_vaghela
SplunkTrust
SplunkTrust

@333_gloom_333

less than 1Mb might be due to partial OR incomplete execution of the search. Can you please try below search. Hopefully, this will give you all the required rows with the performance improvement.

| tstats `summariesonly` count by source, host, index, sourcetype 
| stats values(host) as src_host dc(host) as count by source, index  | eval n=1 | accum n
| sort + count 
| stats values(*) as * by n,src_host | table sourcetype index src_host count
| outputlookup sourcescheck

Thanks

View solution in original post

kamlesh_vaghela
SplunkTrust
SplunkTrust

@333_gloom_333

less than 1Mb might be due to partial OR incomplete execution of the search. Can you please try below search. Hopefully, this will give you all the required rows with the performance improvement.

| tstats `summariesonly` count by source, host, index, sourcetype 
| stats values(host) as src_host dc(host) as count by source, index  | eval n=1 | accum n
| sort + count 
| stats values(*) as * by n,src_host | table sourcetype index src_host count
| outputlookup sourcescheck

Thanks

View solution in original post

starcher
SplunkTrust
SplunkTrust

Do not table then stats. Just use the stats removing the table line in the middle. You are breaking the map reduce behavior efficiencies.

0 Karma

333_gloom_333
Engager

It works. Although i was left disappointed about mvexpand command.

Thanks a lot!

0 Karma

kamlesh_vaghela
SplunkTrust
SplunkTrust

Yes, make sense. Thanks, @starcher 🙂

0 Karma