topic Alternatives to using MVExpand - running into limitations in Splunk Enterprise

Alternatives to using MVExpand - running into limitations

ch1221 — Fri, 26 Feb 2021 14:44:41 GMT

I'm looking for another way to run the search below and expand the computer field. This search is pulling systems belonging to a specific group in AD and then cleaning up the name from the member_dn field. It them puts it into a lookup table to use in ES.
Mvexpand is running into limitations with memory and I cannot adjust it high enough to extract all of the values.

|ldapsearch domain=default search="(&(objectclass=group)(cn=Eng_Computers))" | table cn,distinguishedName | ldapgroup| table cn,member_dn,member_type
| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)" |mvexpand computer |table computer | sort computer |outputlookup eng_systems.csv

Suggestions are appreciated.

Re: Alternatives to using MVExpand - running into limitations

ITWhisperer — Fri, 26 Feb 2021 15:17:19 GMT

Instead of

| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)" |mvexpand computer

try this (a bit tortuous admittedly)

| streamstats count as row | eval steps=mvcount(member_dn) | streamstats sum(steps) as toprow | eval maxrow=toprow | makecontinuous toprow | reverse | filldown | eval toprow=if(row=1,1,toprow) | makecontinuous toprow | filldown | eval member_dn=mvindex(member_dn,maxrow-toprow) | fields - maxrow toprow row steps | rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)"

Re: Alternatives to using MVExpand - running into limitations

ch1221 — Fri, 26 Feb 2021 15:24:25 GMT

unfortunately that only returns one system out of the group.

Re: Alternatives to using MVExpand - running into limitations

ITWhisperer — Fri, 26 Feb 2021 15:27:30 GMT

OK try it the other around

| rex max_match=0 field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)" | streamstats count as row | eval steps=mvcount(computer) | streamstats sum(steps) as toprow | eval maxrow=toprow | makecontinuous toprow | reverse | filldown | eval toprow=if(row=1,1,toprow) | makecontinuous toprow | filldown | eval computer=mvindex(computer,maxrow-toprow) | fields - maxrow toprow row steps

Re: Alternatives to using MVExpand - running into limitations

ch1221 — Fri, 26 Feb 2021 15:32:32 GMT

Still only one system being returned 😞

Re: Alternatives to using MVExpand - running into limitations

ITWhisperer — Fri, 26 Feb 2021 15:36:05 GMT

Can you share some data of the events you have after

|ldapgroup| table cn,member_dn,member_type

Re: Alternatives to using MVExpand - running into limitations

ch1221 — Fri, 26 Feb 2021 16:22:09 GMT

I removed the additional fields so it's just member_dn. Here's a very small sample of the 9,000+

member_dn
CN=ORW-EG-M480,OU=Win7,OU=xxx Workstations,OU=xxx,OU=Amer,DC=xxx,DC=xxx,DC=com
CN=FRG-W10-SCH,OU=Win7,OU=xxx,OU=EMEA,DC=xxx,DC=xxx,DC=com
CN=FRS-MARV-L,OU=Win7,OU=xxx,OU=EMEA,DC=mgc,DC=xxx,DC=com

Re: Alternatives to using MVExpand - running into limitations

ITWhisperer — Fri, 26 Feb 2021 16:25:46 GMT

Is this a multi-value field? Do you get the correct count (in steps) if you do this

| eval steps=mvcount(member_dn)

Re: Alternatives to using MVExpand - running into limitations

ch1221 — Fri, 26 Feb 2021 16:31:41 GMT

Yes, steps returns 9056

Re: Alternatives to using MVExpand - running into limitations

ITWhisperer — Fri, 26 Feb 2021 16:37:26 GMT

So, does this generate enough copies of the events?

Re: Alternatives to using MVExpand - running into limitations

ch1221 — Fri, 26 Feb 2021 16:44:13 GMT

This still only provides the results as a list in 1 event instead of breaking them out.

Re: Alternatives to using MVExpand - running into limitations

ITWhisperer — Fri, 26 Feb 2021 17:21:30 GMT

Do you only have 1 event? If so, that is probably the issue. The expansion works for multiple rows. Try this instead

| streamstats count as row | eval steps=mvcount(member_dn) | streamstats sum(steps) as toprow | eval maxrow=toprow | reverse | append [| makeresults | eval toprow=1 | fields - _time] | reverse | makecontinuous toprow | reverse | filldown | sort toprow | eval member_dn=mvindex(member_dn,maxrow-toprow) | fields - maxrow toprow row steps | rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)"

Re: Alternatives to using MVExpand - running into limitations

ch1221 — Fri, 26 Feb 2021 17:26:06 GMT

Yes, that works!!!! Thank you so much for your help!!!

Re: Alternatives to using MVExpand - running into limitations

drejoe — Wed, 28 Jul 2021 12:02:26 GMT

@ITWhisperer

And when you run into the limitation of 50000 on makecontinuous.!
Any alternatives to this issue?
I've the need of handling quite more than 50000 with a simulare function as makecontinuous.

Any idea?

//T

Re: Alternatives to using MVExpand - running into limitations

ITWhisperer — Wed, 28 Jul 2021 12:06:54 GMT

Try increasing the limit in limits.conf

Re: Alternatives to using MVExpand - running into limitations

drejoe — Wed, 28 Jul 2021 12:22:14 GMT

max_mem_usage_mb under the default stanza or?

Re: Alternatives to using MVExpand - running into limitations

ITWhisperer — Wed, 28 Jul 2021 12:43:44 GMT

To be honest, I don't know. It could be any one or more of these (or something else).

[searchresults]

* This stanza controls search results for a variety of Splunk search commands.

maxresultrows = <integer>
* Configures the maximum number of events are generated by search commands
  which grow the size of your result set (such as multikv) or that create
  events. Other search commands are explicitly controlled in specific stanzas
  below.
* This limit should not exceed 50000.
* Default: 50000

or this

Distributed search

# This section contains settings for distributed search connection
# information.

max_combiner_memevents = <integer>
* Maximum size of the in-memory buffer for the search results combiner.
  The <integer> is the number of events.
* Default: 50000

or this

Results storage

# This section contains settings for storing final search results.

max_count = <integer>
* The number of events that can be accessible in any given status bucket
  (when status_buckets = 0).
* The last accessible event in a call that takes a base and count.
* NOTE: This value does not reflect the number of events displayed in the
  UI after the search is evaluated or computed.
* Default: 500000

or this

[anomalousvalue]

maxresultrows = <integer>
* Configures the maximum number of events that can be present in memory at one
  time.
* Default: The value set for 'maxresultrows' in the [searchresults] stanza,
  which is 50000 by default.