Splunk Enterprise

Alternatives to using MVExpand - running into limitations

ch1221
Path Finder

I'm looking for another way to run the search below and expand the computer field. This search is pulling systems belonging to a specific group in AD and then cleaning up the name from the member_dn field.  It them puts it into a lookup table to use in ES.
Mvexpand is running into limitations with memory and I cannot adjust it high enough to extract all of the values.  

|ldapsearch domain=default search="(&(objectclass=group)(cn=Eng_Computers))" | table cn,distinguishedName | ldapgroup| table cn,member_dn,member_type
| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)" |mvexpand computer |table computer | sort computer |outputlookup eng_systems.csv

Suggestions are appreciated.

 

Labels (1)
Tags (2)
0 Karma
1 Solution

ITWhisperer
SplunkTrust
SplunkTrust

Do you only have 1 event? If so, that is probably the issue. The expansion works for multiple rows. Try this instead

| streamstats count as row
| eval steps=mvcount(member_dn)
| streamstats sum(steps) as toprow
| eval maxrow=toprow
| reverse
| append [| makeresults | eval toprow=1 | fields - _time]
| reverse
| makecontinuous toprow
| reverse
| filldown
| sort toprow
| eval member_dn=mvindex(member_dn,maxrow-toprow)
| fields - maxrow toprow row steps
| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)"

 

View solution in original post

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Instead of

| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)" |mvexpand computer

try this (a bit tortuous admittedly)

| streamstats count as row
| eval steps=mvcount(member_dn)
| streamstats sum(steps) as toprow
| eval maxrow=toprow
| makecontinuous toprow
| reverse
| filldown
| eval toprow=if(row=1,1,toprow)
| makecontinuous toprow
| filldown
| eval member_dn=mvindex(member_dn,maxrow-toprow)
| fields - maxrow toprow row steps
| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)"

 

0 Karma

ch1221
Path Finder

unfortunately that only returns one system out of the group.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

OK try it the other around

| rex max_match=0 field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)"
| streamstats count as row
| eval steps=mvcount(computer)
| streamstats sum(steps) as toprow
| eval maxrow=toprow
| makecontinuous toprow
| reverse
| filldown
| eval toprow=if(row=1,1,toprow)
| makecontinuous toprow
| filldown
| eval computer=mvindex(computer,maxrow-toprow)
| fields - maxrow toprow row steps
0 Karma

ch1221
Path Finder

Still only one system being returned 😞

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Can you share some data of the events you have after

|ldapgroup| table cn,member_dn,member_type
0 Karma

ch1221
Path Finder

I removed the additional fields so it's just member_dn.  Here's a very small sample of the 9,000+

member_dn
CN=ORW-EG-M480,OU=Win7,OU=xxx Workstations,OU=xxx,OU=Amer,DC=xxx,DC=xxx,DC=com
CN=FRG-W10-SCH,OU=Win7,OU=xxx,OU=EMEA,DC=xxx,DC=xxx,DC=com
CN=FRS-MARV-L,OU=Win7,OU=xxx,OU=EMEA,DC=mgc,DC=xxx,DC=com

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Is this a multi-value field? Do you get the correct count (in steps) if you do this

| eval steps=mvcount(member_dn)
0 Karma

ch1221
Path Finder

Yes, steps returns 9056

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

So, does this generate enough copies of the events?

| streamstats count as row
| eval steps=mvcount(member_dn)
| streamstats sum(steps) as toprow
| eval maxrow=toprow
| makecontinuous toprow
| reverse
| filldown
| eval toprow=if(row=1,1,toprow)
| makecontinuous toprow
| filldown
0 Karma

ch1221
Path Finder

This still only provides the results as a list in 1 event instead of breaking them out.

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Do you only have 1 event? If so, that is probably the issue. The expansion works for multiple rows. Try this instead

| streamstats count as row
| eval steps=mvcount(member_dn)
| streamstats sum(steps) as toprow
| eval maxrow=toprow
| reverse
| append [| makeresults | eval toprow=1 | fields - _time]
| reverse
| makecontinuous toprow
| reverse
| filldown
| sort toprow
| eval member_dn=mvindex(member_dn,maxrow-toprow)
| fields - maxrow toprow row steps
| rex field=member_dn "CN\=(?P<computer>[\w\-\_]+)(?=\,\w{2}\=)"

 

0 Karma

drejoe
Explorer

@ITWhisperer 

And when you run into the limitation of 50000 on makecontinuous.!
Any alternatives to this issue?
I've the need of handling quite more than 50000 with a simulare function as makecontinuous.

Any idea?

//T

0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

Try increasing the limit in limits.conf

0 Karma

drejoe
Explorer
max_mem_usage_mb under the default stanza or?
0 Karma

ITWhisperer
SplunkTrust
SplunkTrust

To be honest, I don't know. It could be any one or more of these (or something else).

[searchresults]

* This stanza controls search results for a variety of Splunk search commands.

maxresultrows = <integer>
* Configures the maximum number of events are generated by search commands
  which grow the size of your result set (such as multikv) or that create
  events. Other search commands are explicitly controlled in specific stanzas
  below.
* This limit should not exceed 50000.
* Default: 50000

or this

Distributed search

# This section contains settings for distributed search connection
# information.

max_combiner_memevents = <integer>
* Maximum size of the in-memory buffer for the search results combiner.
  The <integer> is the number of events.
* Default: 50000

 or this

Results storage

# This section contains settings for storing final search results.

max_count = <integer>
* The number of events that can be accessible in any given status bucket
  (when status_buckets = 0).
* The last accessible event in a call that takes a base and count.
* NOTE: This value does not reflect the number of events displayed in the
  UI after the search is evaluated or computed.
* Default: 500000

or this

[anomalousvalue]

maxresultrows = <integer>
* Configures the maximum number of events that can be present in memory at one
  time.
* Default: The value set for 'maxresultrows' in the [searchresults] stanza,
  which is 50000 by default.
  

 

0 Karma

ch1221
Path Finder

Yes, that works!!!! Thank you so much for your help!!!

0 Karma
Get Updates on the Splunk Community!

Routing Data to Different Splunk Indexes in the OpenTelemetry Collector

This blog post is part of an ongoing series on OpenTelemetry. The OpenTelemetry project is the second largest ...

Getting Started with AIOps: Event Correlation Basics and Alert Storm Detection in ...

Getting Started with AIOps:Event Correlation Basics and Alert Storm Detection in Splunk IT Service ...

Register to Attend BSides SPL 2022 - It's all Happening October 18!

Join like-minded individuals for technical sessions on everything Splunk!  This is a community-led and run ...