I give my splunk 50GB Mem with
max_mem_usage_mb = 50480
in the limits.conf
but splunk 5.0.3 gives me a "mvexpand output will be truncated due to excessive memory usage".
THe job inspector shows that the incoming data are a few 10 MB.
Miss I a hidden config-option?
Best regards Marco
Hi Frank,
in my case I have solved it. I suggest to remove alle fields you don't need anymore, before you call the mvexpand, like ... | fields dns, ip, record | fields - _raw | ...
Perhaps this helps.
By Marco
Hi Frank,
in my case I have solved it. I suggest to remove alle fields you don't need anymore, before you call the mvexpand, like ... | fields dns, ip, record | fields - _raw | ...
Perhaps this helps.
By Marco
still works in 2020
Another workaround is to use the "by" clause of a stats command to split a multivalued field into its values, it doesn't have the memory issue that mvexpand has...
|makeresults count=10000
|streamstats count as t
|stats values(t) as multivalue
|fields multivalue
`comment("multivalue extract without mvexpand")`
|stats count by multivalue
|table multivalue
All multi-values must be unique values, but that's OK
still Octuber 2019 🙂
June 2019 🙂
Still a good answer in 2019.
Tks man, "fields - _raw" fixed my problem
This is still an excellent answer on 2018
Still a good answer in 2017.
Thanks, that fixed it.
Hi Marco,
I'm seeing the same behavior in 5.0.3. Did you ever find a solution? I'm trying to get some info from a REST input into a lookup, and the seemingly inefficient technique in the docs for spath to combine multivalued fields bombs at exactly 700 elements. Anybody else seeing this, or have any ideas?
-Frank
EDIT - Upgrading to 5.0.4 and greatly increasing max_mem_usage_mb did not resolve the problem for me but filtering out the _raw fields as Marco suggested did. My working query:
index=rest sourcetype=dns:rest:a | head 1
| spath output=dns path={}.name
| spath output=ip path={}.ipv4addr
| fields - _raw
| eval record=mvzip(dns,ip)
| fields + record
| mvexpand record | eval record = split(record,",")
| eval dns=mvindex(record,0) | eval ip=mvindex(record,1)
| table dns,ip
Hi Frank,
another solution may be to incease the max_mem_usage_mb in your limits.conf?
You can avoid the spath in your query, by defining it under
Manager » Fields » Field aliases
best regards
Marco