Help with regex search on multiple lines and displaying only if not matching a particular string

[tha_ghost99]

Below is the current out put (raw) - specific field

node0:
--------------------------------------------------------------------------
/var/: No such file or directory
/var/tmp/: No such file or directory
/var/: blablablaba.txt

node1:
--------------------------------------------------------------------------
/var/: No such file or directory
/var/tmp/: No such file or directory

what i need help on, is to group Node0 and Node1 as their own group, and only show IF the row below it (after the "/var") if its anything BUT "No such file or directory"

so the output will end up being:

NODE0:

/var/: blablablaba.txt

NODE1:

thanks for the help in advance.

[yuanliu]

Assuming the field name is data, you can do

| eval data = replace(data, "\n(.*: No such file or directory|---*)", "")

data

node0: /var/: blablablaba.txt

node1:

[tha_ghost99]

hi @yuanliu  thanks for the quick reply.

i think i need more help to further clarify.

so this is the Raw data (but splunk under verbose mode, it was able to consider this as a field named "core dumps"

node0:
--------------------------------------------------------------------------
/var/: No such file or directory
/var/tmp/: No such file or directory
/var/: blablablaba.txt

node1:
--------------------------------------------------------------------------
/var/: No such file or directory
/var/tmp/: No such file or directory

what i would like to do, is do a rex max_match where it would create values that would look like this first.

FIRST OUTPUT:

node0,/var/: No such file or directory
node0,/var/tmp/: No such file or directory
node0,/var/: blablablaba.txt

node1,/var/: No such file or directory
node1,/var/tmp/: No such file or directory

from here i would like to then display all lines that do not have "No such file or directory"

so the final output will be:

node0,/var/: blablablaba.txt

i hope this makes it more clearer. i do apologize for making it confusing.

[yuanliu]

I mistook node0 and node1 as separate event.  But no matter.  You can use the same approach: Just split the two components into separate events. 

| eval core_dump = split(core_dump, "

")
| mvexpand core_dump
| eval core_dump = replace(core_dump, "\n(.*: No such file or directory|---*)", "")

If you don't want to display the node that has no file,  add a filter, like

| eval core_dump = split(core_dump, "

")
| mvexpand core_dump
| eval core_dump = replace(core_dump, "\n(.*: No such file or directory|---*)", "")
| where match(core_dump, ": *\w+")

[tha_ghost99]

@yuanliu  hi appreciate the help but didnt seem to work.

can i bug you for an easier but similar help? maybe it will help me better understand since i am a novice with splunks.

here is my data:

node0:
/hello
/hi
-rw-rw---- bad1.gz
-rw-rw---- bad2.gz
/bye

node1:
/hello/hello
/hi/hi
-rw-rw---- bad3.gz
/bye/bye

how can i simply group them based on node0 and node1? using rex and mvzip for example or some other variations.

output would be:

node0:/hello
node0:/hi
node0:-rw-rw---- bad1.gz
node0:-rw-rw---- bad2.gz
node0:/bye
node1,/hello/hello
node1,/hi/hi
node1,-rw-rw---- bad3.gz
node1,/bye/bye

hope this helps. 🙂 again really thank you

[yuanliu]

Let me demonstrate from your original dataset (which is not really different from this simplified set) first because that's exactly how I came to the code.  You can follow along by copying into your search window (and compare data and data flow with your actual data, check my assumptions).  I will also link to official documentation of key commands/functions.

core_dump

node0: -------------------------------------------------------------------------- /var/: No such file or directory /var/tmp/: No such file or directory /var/: blablablaba.txt node1: -------------------------------------------------------------------------- /var/: No such file or directory /var/tmp/: No such file or directory

Obviously if the above data are considerably different from the real one, the code will not give expected result in the real world.  (For example: Is there a line completely made of dashes?)  The above is simulated using the following:

| makeresults
| eval core_dump = "node0:
--------------------------------------------------------------------------
/var/: No such file or directory
/var/tmp/: No such file or directory
/var/: blablablaba.txt

node1:
--------------------------------------------------------------------------
/var/: No such file or directory
/var/tmp/: No such file or directory"
| fields - _time
``` data emulation above ```

Now, split core_dump after data emulation

| eval core_dump = split(core_dump, "

") ``` splits two nodes into multivalue ```

Note: There is a blank line between the two quotation marks; there is no trailing space after the first quotation mark, and no leading space before the second one.  This does not seem to do much from a first glance, but core_dump is now multivalue.  Next, add mvexpand.

| mvexpand core_dump

The search results now look like thus:

core_dump

node0: -------------------------------------------------------------------------- /var/: No such file or directory /var/tmp/: No such file or directory /var/: blablablaba.txt

node1: -------------------------------------------------------------------------- /var/: No such file or directory /var/tmp/: No such file or directory

In other words, the two nodes are in their own events.

Next, do some regex magic with replace.

| eval core_dump = replace(core_dump, "\n(.*: No such file or directory|---*)", "") ``` vanish unwanted lines ```

you get

core_dump

node0: /var/: blablablaba.txt

node1:

Lastly, filter out nodes that do not have real files using where.

| where match(core_dump, ": *\w+")

The end result is

core_dump

node0: /var/: blablablaba.txt

As you can see, every step in the above is moving toward your initial mockup of desired output.  And it matches your mockup exactly.

After you changed requirement

Now, you changed your desired output, so we will do a little more. (As is always in SPL, code depends tremendously on input data as well as desired output.)  You also changed your input to remove that line composed of all dashes.  So emulation will be a little different.  Let's add in node split, and mvexpand as a start point.

| makeresults
| eval core_dump = "node0:
/var/: No such file or directory
/var/tmp/: No such file or directory
/var/: blablablaba.txt

node1:
/var/: No such file or directory
/var/tmp/: No such file or directory"
| fields - _time
``` data emulation above, no ---- ```
| eval core_dump = split(core_dump, "

") ``` split two nodes ```
| mvexpand core_dump ``` split events according to node ```

core_dump

node0: /var/: No such file or directory /var/tmp/: No such file or directory /var/: blablablaba.txt

node1: /var/: No such file or directory /var/tmp/: No such file or directory

After this, run split again

| eval core_dump = split(core_dump, "
") ``` split by line ```

(Again, you may not see the difference with a casual glance.  But if you run mvcount before and after split, you'll see the difference.)  Next, render this one field into two separate fields:

| eval node = mvindex(core_dump, 0)
| eval core_dump = mvindex(core_dump, 1, mvcount(core_dump))

core_dump	node
/var/: No such file or directory /var/tmp/: No such file or directory /var/: blablablaba.txt	node0:
/var/: No such file or directory /var/tmp/: No such file or directory	node1:

Finally, smooch the two fields to make one final display:

| eval core_dump = mvmap(core_dump, node . core_dump) ``` print the desired pattern ```
| fields - node

(mvmap is needed because a string cannot be added to a multivalue field even if each value is a string.)

core_dump

node0:/var/: No such file or directory node0:/var/tmp/: No such file or directory node0:/var/: blablablaba.txt

node1:/var/: No such file or directory node1:/var/tmp/: No such file or directory

Does this help?

[tha_ghost99]

hi @yuanliu 

i was able to follow it when the data is from what i pasted. here is the raw output. i was wondering if you can work your magic, it just dont work for me when i use the raw data.

Oct 11 11:32:45
node0:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
-rw-rw---- 1 nobody wheel 1251399200 Oct 11 10:29 /var/tmp/C1.core.0.gz
-rw-rw---- 1 nobody wheel 1254654282 Oct 11 10:27 /var/tmp/C3.core.0.gz
-rw-rw---- 1 nobody wheel 1093812660 Mar 11 2022 /var/tmp/C0.core.0.gz
-rw-rw---- 1 nobody wheel 1255530083 Oct 11 10:24 /var/tmp/IC1.core.0.gz
-rw-rw---- 1 nobody wheel 184401920 Mar 11 2022 /var/tmp/IC2.core.0.gz
-rw-rw---- 1 nobody wheel 1251262474 Mar 1 2022 /var/tmp/IC3.core.0.gz
-rw-rw---- 1 nobody wheel 1256350152 Oct 12 10:21 /var/tmp/IC3.core.1.gz
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/var/jails/rest-api/tmp/*core*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
/jail/var/tmp/*core*: No such file or directory
total files: 7

node1:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
-rw-rw---- 1 nobody wheel 34766848 Mar 2 2022 /var/tmp/IC3.core.0.gz
-rw-rw---- 1 nobody wheel 1151312299 Mar 1 2022 /var/tmp/IC0.core.0.gz
-rw-rw---- 1 nobody wheel 1153074056 Mar 1 2022 /var/tmp/IC1.core.0.gz
-rw-rw---- 1 nobody wheel 1158385356 Mar 1 2022 /var/tmp/IC2.core.0.gz
-rw-rw---- 1 nobody wheel 1150786252 Mar 1 2022 /var/tmp/IC3.core.0.gz
-rw-rw---- 1 root wheel 140366826 Dec 14 2021 /var/tmp/2d.core-tarball.0.tgz
-rw-rw---- 1 root wheel 140020614 Feb 12 2022 /var/tmp/b2d.core-tarball.1.tgz
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/var/jails/rest-api/tmp/*core*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
/jail/var/tmp/*core*: No such file or directory
total files: 7

{primary:node0}

[yuanliu]

You previously stated that you have a field called "core_dump" that only contain the dump data.  To work from _raw, you need to get to that field first.

| rex mode=sed "s/.*//" ``` get rid of first line ```
| eval core_dump = mvindex(split(_raw, "

"), 0, 1) ``` split two nodes ```
| mvexpand core_dump ``` split events according to node ```
| eval core_dump = split(core_dump, "
")
| rex mode=sed field=core_dump "s/(.*: No such file or directory|---*|total files:.*)//" ``` remove unwanted lines ```

(Somehow replace doesn't work with this set of data.  But rex still works.)

Here is data emulation you can compare against.

| makeresults
| eval _raw = "Oct 11 11:32:45
node0:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
-rw-rw---- 1 nobody wheel 1251399200 Oct 11 10:29 /var/tmp/C1.core.0.gz
-rw-rw---- 1 nobody wheel 1254654282 Oct 11 10:27 /var/tmp/C3.core.0.gz
-rw-rw---- 1 nobody wheel 1093812660 Mar 11 2022 /var/tmp/C0.core.0.gz
-rw-rw---- 1 nobody wheel 1255530083 Oct 11 10:24 /var/tmp/IC1.core.0.gz
-rw-rw---- 1 nobody wheel 184401920 Mar 11 2022 /var/tmp/IC2.core.0.gz
-rw-rw---- 1 nobody wheel 1251262474 Mar 1 2022 /var/tmp/IC3.core.0.gz
-rw-rw---- 1 nobody wheel 1256350152 Oct 12 10:21 /var/tmp/IC3.core.1.gz
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/var/jails/rest-api/tmp/*core*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
/jail/var/tmp/*core*: No such file or directory
total files: 7

node1:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
-rw-rw---- 1 nobody wheel 34766848 Mar 2 2022 /var/tmp/IC3.core.0.gz
-rw-rw---- 1 nobody wheel 1151312299 Mar 1 2022 /var/tmp/IC0.core.0.gz
-rw-rw---- 1 nobody wheel 1153074056 Mar 1 2022 /var/tmp/IC1.core.0.gz
-rw-rw---- 1 nobody wheel 1158385356 Mar 1 2022 /var/tmp/IC2.core.0.gz
-rw-rw---- 1 nobody wheel 1150786252 Mar 1 2022 /var/tmp/IC3.core.0.gz
-rw-rw---- 1 root wheel 140366826 Dec 14 2021 /var/tmp/2d.core-tarball.0.tgz
-rw-rw---- 1 root wheel 140020614 Feb 12 2022 /var/tmp/b2d.core-tarball.1.tgz
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/var/jails/rest-api/tmp/*core*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
/jail/var/tmp/*core*: No such file or directory
total files: 7

{primary:node0}"
| fields - _time
``` data emulation above ```

[yuanliu · Accepted solution]

You previously stated that you have a field called "core_dump" that only contain the dump data.  To work from _raw, you need to get to that field first.

---

| rex mode=sed "s/.*//" ``` get rid of first line ```
| eval core_dump = mvindex(split(_raw, "

"), 0, 1) ``` split two nodes ```
| mvexpand core_dump ``` split events according to node ```
| eval core_dump = mvmap(core_dump, replace(core_dump, "\n(.*: No such file or directory|---*|total files:.*)", "")) ``` remove unwanted lines ```

(I assume that you don't want that total files line, either.)

Again, I tested against this emulation.

| makeresults
| eval _raw = "Oct 11 11:32:45
node0:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
-rw-rw---- 1 nobody wheel 1251399200 Oct 11 10:29 /var/tmp/C1.core.0.gz
-rw-rw---- 1 nobody wheel 1254654282 Oct 11 10:27 /var/tmp/C3.core.0.gz
-rw-rw---- 1 nobody wheel 1093812660 Mar 11 2022 /var/tmp/C0.core.0.gz
-rw-rw---- 1 nobody wheel 1255530083 Oct 11 10:24 /var/tmp/IC1.core.0.gz
-rw-rw---- 1 nobody wheel 184401920 Mar 11 2022 /var/tmp/IC2.core.0.gz
-rw-rw---- 1 nobody wheel 1251262474 Mar 1 2022 /var/tmp/IC3.core.0.gz
-rw-rw---- 1 nobody wheel 1256350152 Oct 12 10:21 /var/tmp/IC3.core.1.gz
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/var/jails/rest-api/tmp/*core*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
/jail/var/tmp/*core*: No such file or directory
total files: 7

node1:
--------------------------------------------------------------------------
/var/crash/*core*: No such file or directory
-rw-rw---- 1 nobody wheel 34766848 Mar 2 2022 /var/tmp/IC3.core.0.gz
-rw-rw---- 1 nobody wheel 1151312299 Mar 1 2022 /var/tmp/IC0.core.0.gz
-rw-rw---- 1 nobody wheel 1153074056 Mar 1 2022 /var/tmp/IC1.core.0.gz
-rw-rw---- 1 nobody wheel 1158385356 Mar 1 2022 /var/tmp/IC2.core.0.gz
-rw-rw---- 1 nobody wheel 1150786252 Mar 1 2022 /var/tmp/IC3.core.0.gz
-rw-rw---- 1 root wheel 140366826 Dec 14 2021 /var/tmp/2d.core-tarball.0.tgz
-rw-rw---- 1 root wheel 140020614 Feb 12 2022 /var/tmp/b2d.core-tarball.1.tgz
/var/tmp/pics/*core*: No such file or directory
/var/crash/kernel.*: No such file or directory
/var/jails/rest-api/tmp/*core*: No such file or directory
/tftpboot/corefiles/*core*: No such file or directory
/jail/var/tmp/*core*: No such file or directory
total files: 7

{primary:node0}"
| fields - _time
``` data emulation above ```

[tha_ghost99]

this is my current search but it does not seem to get the other lines below..

its only checking 1 line from each group of node0 and node1 😞

|data
| rex field="core_dump" max_match=0 "(?s)(?<nodeNum>node\d+):.*?------------------------*[\n](?<coredumpoutput>[^\n]*)"
|table nodeNum coredumpoutput

output:

nodeNum coredumpoutput

node0 node1

/var/crash/*core*: No such file or directory /var/crash/*core*: No such file or directory