mvexpand gives "mvexpand output will be truncated due to excessive memory usage"

[marcokrueger]

I give my splunk 50GB Mem with
max_mem_usage_mb = 50480
in the limits.conf
but splunk 5.0.3 gives me a "mvexpand output will be truncated due to excessive memory usage".
THe job inspector shows that the incoming data are a few 10 MB.

Miss I a hidden config-option?

Best regards Marco

[marcokrueger · Accepted solution]

Hi Frank,
in my case I have solved it. I suggest to remove alle fields you don't need anymore, before you call the mvexpand, like ... | fields dns, ip, record | fields - _raw | ...
Perhaps this helps.

By Marco

[ClubMed]

Wow.

For all my queries, I had been using the following fields command under the assumption it did drop _raw.

| fields _time, xxx, yyy, zzz, ....

Then one day I started a large mvexpand and ran into memory limit.

My thought upon seeing this was 'Huh? Well, worth a try I guess.'

| fields _time, xxx, yyy, zzz, ....
| fields - _raw

Boom, mvexpand completes successfully. The heck? It actually cut the search time in half too.

[amckinnie_splun]

still works in 2020

[gjanders]

Another workaround is to use the "by" clause of a stats command to split a multivalued field into its values, it doesn't have the memory issue that mvexpand has...

[to4kawa]

|makeresults count=10000
|streamstats count as t
|stats values(t) as multivalue 
|fields multivalue 
`comment("multivalue extract without mvexpand")`
|stats count by multivalue 
|table multivalue 

All multi-values must be unique values, but that's OK

[LM_ACN]

still Octuber 2019 🙂

[sgundeti573]

June 2019 🙂

[balmeida]

Still a good answer in 2019.

[rafaelvjb]

Tks man, "fields - _raw" fixed my problem

[jstubberfield]

This is still an excellent answer on 2018

[grittonc]

Still a good answer in 2017.

[fervin]

Thanks, that fixed it.

[fervin]

Hi Marco,

I'm seeing the same behavior in 5.0.3. Did you ever find a solution? I'm trying to get some info from a REST input into a lookup, and the seemingly inefficient technique in the docs for spath to combine multivalued fields bombs at exactly 700 elements. Anybody else seeing this, or have any ideas?

-Frank

EDIT - Upgrading to 5.0.4 and greatly increasing max_mem_usage_mb did not resolve the problem for me but filtering out the _raw fields as Marco suggested did. My working query:

index=rest sourcetype=dns:rest:a | head 1 
| spath output=dns path={}.name 
| spath output=ip path={}.ipv4addr
| fields - _raw 
| eval record=mvzip(dns,ip)
| fields + record
| mvexpand record | eval record = split(record,",") 
| eval dns=mvindex(record,0) | eval ip=mvindex(record,1)   
| table dns,ip

[marcokrueger]

Hi Frank,
another solution may be to incease the max_mem_usage_mb in your limits.conf?

You can avoid the spath in your query, by defining it under
Manager » Fields » Field aliases

best regards
Marco