Alternatives to using MVExpand - running into limitations

[ch1221]

I'm looking for another way to run the search below and expand the computer field. This search is pulling systems belonging to a specific group in AD and then cleaning up the name from the member_dn field.  It them puts it into a lookup table to use in ES.
Mvexpand is running into limitations with memory and I cannot adjust it high enough to extract all of the values.  

|ldapsearch domain=default search="(&(objectclass=group)(cn=Eng_Computers))" | table cn,distinguishedName | ldapgroup| table cn,member_dn,member_type
| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)" |mvexpand computer |table computer | sort computer |outputlookup eng_systems.csv

Suggestions are appreciated.

[ITWhisperer]

Instead of

| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)" |mvexpand computer

try this (a bit tortuous admittedly)

| streamstats count as row
| eval steps=mvcount(member_dn)
| streamstats sum(steps) as toprow
| eval maxrow=toprow
| makecontinuous toprow
| reverse
| filldown
| eval toprow=if(row=1,1,toprow)
| makecontinuous toprow
| filldown
| eval member_dn=mvindex(member_dn,maxrow-toprow)
| fields - maxrow toprow row steps
| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)"

[ch1221]

unfortunately that only returns one system out of the group.

[ITWhisperer]

OK try it the other around

| rex max_match=0 field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)"
| streamstats count as row
| eval steps=mvcount(computer)
| streamstats sum(steps) as toprow
| eval maxrow=toprow
| makecontinuous toprow
| reverse
| filldown
| eval toprow=if(row=1,1,toprow)
| makecontinuous toprow
| filldown
| eval computer=mvindex(computer,maxrow-toprow)
| fields - maxrow toprow row steps

[ch1221]

Still only one system being returned 😞

[ITWhisperer]

Can you share some data of the events you have after

|ldapgroup| table cn,member_dn,member_type

[ch1221]

I removed the additional fields so it's just member_dn.  Here's a very small sample of the 9,000+

member_dn
CN=ORW-EG-M480,OU=Win7,OU=xxx Workstations,OU=xxx,OU=Amer,DC=xxx,DC=xxx,DC=com
CN=FRG-W10-SCH,OU=Win7,OU=xxx,OU=EMEA,DC=xxx,DC=xxx,DC=com
CN=FRS-MARV-L,OU=Win7,OU=xxx,OU=EMEA,DC=mgc,DC=xxx,DC=com

[ITWhisperer]

Is this a multi-value field? Do you get the correct count (in steps) if you do this

| eval steps=mvcount(member_dn)

[ch1221]

Yes, steps returns 9056

[ITWhisperer]

So, does this generate enough copies of the events?

| streamstats count as row
| eval steps=mvcount(member_dn)
| streamstats sum(steps) as toprow
| eval maxrow=toprow
| makecontinuous toprow
| reverse
| filldown
| eval toprow=if(row=1,1,toprow)
| makecontinuous toprow
| filldown

[ch1221]

This still only provides the results as a list in 1 event instead of breaking them out.

[ITWhisperer · Accepted solution]

Do you only have 1 event? If so, that is probably the issue. The expansion works for multiple rows. Try this instead

| streamstats count as row
| eval steps=mvcount(member_dn)
| streamstats sum(steps) as toprow
| eval maxrow=toprow
| reverse
| append [| makeresults | eval toprow=1 | fields - _time]
| reverse
| makecontinuous toprow
| reverse
| filldown
| sort toprow
| eval member_dn=mvindex(member_dn,maxrow-toprow)
| fields - maxrow toprow row steps
| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)"

[drejoe]

@ITWhisperer 

And when you run into the limitation of 50000 on makecontinuous.!
Any alternatives to this issue?
I've the need of handling quite more than 50000 with a simulare function as makecontinuous.

Any idea?

//T

[ITWhisperer]

Try increasing the limit in limits.conf

[drejoe]

max_mem_usage_mb under the default stanza or?

[ITWhisperer]

To be honest, I don't know. It could be any one or more of these (or something else).

[searchresults]

* This stanza controls search results for a variety of Splunk search commands.

maxresultrows = <integer>
* Configures the maximum number of events are generated by search commands
  which grow the size of your result set (such as multikv) or that create
  events. Other search commands are explicitly controlled in specific stanzas
  below.
* This limit should not exceed 50000.
* Default: 50000

or this

Distributed search

# This section contains settings for distributed search connection
# information.

max_combiner_memevents = <integer>
* Maximum size of the in-memory buffer for the search results combiner.
  The <integer> is the number of events.
* Default: 50000

 or this

Results storage

# This section contains settings for storing final search results.

max_count = <integer>
* The number of events that can be accessible in any given status bucket
  (when status_buckets = 0).
* The last accessible event in a call that takes a base and count.
* NOTE: This value does not reflect the number of events displayed in the
  UI after the search is evaluated or computed.
* Default: 500000

or this

[anomalousvalue]

maxresultrows = <integer>
* Configures the maximum number of events that can be present in memory at one
  time.
* Default: The value set for 'maxresultrows' in the [searchresults] stanza,
  which is 50000 by default.
  

[ch1221]

Yes, that works!!!! Thank you so much for your help!!!