topic Re: What are search command alternatives to mvxpand for expansion with filter? in Getting Data In

What are search command alternatives to mvxpand for expansion with filter?

dtakacssplunk — Fri, 01 Dec 2017 05:33:47 GMT

I have a log line logically looking something like this:

f1=a f2=b f3=c custom=[]{ c1{f=x} c2{f=y} c3{f=x}}

I need to count the number of c sections where f=x, for example in above it's 2.

I'm getting quite a lot of errors while using mvexpand: "command.mvexpand: output will be truncated at 6100 results due to excessive memory usage. Memory threshold of 500MB has been reached" due to this I believe there the result counts are not accrurate.

I have removed the all fields including _raw before mvexpand (which helped a lot) but still mvexpand errors come up.

How have people solved this in past? will any of the bellow work?
- run the script on smaller time range then aggregate it outside of splunk
- use alternative ways instead of mvexpand to do expansion with filter?

thx...

Re: What are search command alternatives to mvxpand for expansion with filter?

niketn — Fri, 01 Dec 2017 06:53:21 GMT

@dtakacssplunk, you will have to provide a better sample of your data. For c section to be multivalued field you should have c and not c1, c2 etc. If it is jSON then you should provide syntactically correct structure, better anonymize data rather than mocking so that data pattern can be anticipated.

Can you try mvfilter() for specific value in multivalued filed and then perform mvcount()?

If not you can also try to use rex with max_match=0 to extract only the field you are interested in. However, for us to assist you would need to provide mocked anonymized data since regular expression is based on pattern matching.

Re: What are search command alternatives to mvxpand for expansion with filter?

DalJeanis — Fri, 01 Dec 2017 22:08:35 GMT

Actually, your question and the problem don't seem to be related. You haven't told us the method you attempted to use to count the sections, nor how f and x are determined. For example, do you need a sum of how many f's are x, and how many are y? Here's a quick and dirty version that should do that without any fuss for up to a couple hundred thousand records.

your base search...
| rex "{(?<key>[^={]+)=(?<value>[^}]+)}" max_match=0
| eval keyvalue=mvzip(key,value,"=")
| streamstats count as recno
| table recno keyvalue 
| mvexpand keyvalue
| rex "^(?<key>[^=]+)=(?<value>.+)$"  
| stats count by recno key value

Re: What are search command alternatives to mvxpand for expansion with filter?

woodcock — Fri, 01 Dec 2017 22:51:13 GMT

Show us your search.