topic Re: Why is my regex not matching for a multivalue field? in Splunk Search

Why is my regex not matching for a multivalue field?

spike021 — Thu, 21 Jan 2016 01:30:18 GMT

I looked through quite a few posts on here and couldn't find an appropriate answer, so please bare with me.

I have events coming into Splunk in JSON format. The top-level fields are extracted fine. However, a nested map/dictionary is giving me issues. When I run a search to get the values from that inner dictionary, it works in that I get a resulting table like:

 A       B
---     ---   
 x       y
         z
         y
         z

 s       m
         n

 u       -  (- means None)

So, the y and z both belong to x and occasionally there are more than 2 items per each x. This happens for any x in A.

Since the cell in the table makes the values in B look separated by a newline, I created a regular expression that I've verified to correctly grab the logical groups for each y and z, if, for instance, they were just in a text box like this:

y
z
y
z
y
z

So the regex would properly grab the two as many times necessary, separately.

What I want to do is pull out each pair and separate the two items into two new fields, say C and D, and then later have a table where I have C and D grouped to field A.

The regex part of the command:

rex field="A{}{}"  "(?<C>[\da-z\.-]+\.[a-z\.]{2,6})\n(?<D>\d{1,3})"

Note: the A{}{} together makes up the multivalue field, B, and A is just A as in the earlier part of my example.

The issue I'm running into is that when I pipe what should be the output from that statement into the table command, I don't get anything.. The regex is definitely confirmed working on a site like http://regexr.com/ just for sanity-checking.

So there must be something I'm missing. Maybe the initial table with my example just looks like newlines separate the two values into rows when it doesn't. In which case I tried using a \s as the separator rather than \n and it still doesn't work.

Or maybe there's a super simple explanation for an obvious mistake I'm making.

Regardless I would appreciate some help very much.

Thanks in advance.

Re: Why is my regex not matching for a multivalue field?

jkat54 — Thu, 21 Jan 2016 12:49:52 GMT

Please provide a full search or at least the table command you are using.

Re: Why is my regex not matching for a multivalue field?

alemarzu — Thu, 21 Jan 2016 14:15:40 GMT

Hi spike,

Did you try using max_match=0, which means unlimited matches ?

| rex max_match=0 field="A"  "(?<C>[\da-z\.-]+\.[a-z\.]{2,6})\n(?<D>\d{1,3})"

Notice that I've also removed the curly brackets.

Re: Why is my regex not matching for a multivalue field?

spike021 — Thu, 21 Jan 2016 15:23:48 GMT

I hadn't before, so I just tried that but I still don't seem to be getting any results. My full search query winds up looking like: index="myindex" | rex max_match=0 field="A" "(?[\da-z\.-]+\.[a-z\.]{2,6})\n(?\d{1,3})" | table "A", "C", "D" so nothing absurdly complicated here just trying to get things working correctly.

Re: Why is my regex not matching for a multivalue field?

spike021 — Thu, 21 Jan 2016 15:24:48 GMT

Mentioned it below, but it looks something like: index="myindex" | rex max_match=0 field="A" "(?[\da-z\.-]+\.[a-z\.]{2,6})\n(?\d{1,3})" | table "A", "C", "D"

So nothing particularly complicated, just to get data output, which isn't happening at all yet.

Re: Why is my regex not matching for a multivalue field?

alemarzu — Thu, 21 Jan 2016 15:38:19 GMT

I just realized that the regex you gave us has an invalid structure. Do you mind sharing a sample data so I can build the proper regex ?

Re: Why is my regex not matching for a multivalue field?

jkat54 — Thu, 21 Jan 2016 15:45:40 GMT

   index="myindex"| rex max_match=0 field="A"  "(?<C>[\da-z\.-]+\.[a-z\.]{2,6})\n(?<D>\d{1,3})" | table "A", "C", "D"

Re: Why is my regex not matching for a multivalue field?

javiergn — Thu, 21 Jan 2016 15:47:21 GMT

What about?

index="myindex" 
| mvexpand A
| rex max_match=0 field="A" "(?mi)(?<C>[\da-z\.-]+\.[a-z\.]{2,6})\n(?<D>\d{1,3})" 
| table "A", "C", "D"

index="myindex" 
| mvexpand A
| rex max_match=0 field="A" "(?si)(?<C>[\da-z\.-]+\.[a-z\.]{2,6})\s+(?<D>\d{1,3})" 
| table "A", "C", "D"

Note the (?mi) and the (?si). More info here: http://www.regular-expressions.info/modifiers.html

Re: Why is my regex not matching for a multivalue field?

jkat54 — Thu, 21 Jan 2016 15:47:58 GMT

It only turns invalid when he quotes like this versus

  like this<><\><><><><><><><

But yes, please provide a sample event.

Re: Why is my regex not matching for a multivalue field?

alemarzu — Thu, 21 Jan 2016 15:50:21 GMT

U were right, thx 😉

Re: Why is my regex not matching for a multivalue field?

jkat54 — Tue, 29 Sep 2020 08:29:49 GMT

So you're looking for a new line in field A? whats the \n for? Are you turning the JSON into one large event using should_linemerge=true? Are you using KV_MODE=JSON? A sample event and your props/transforms would be most helpful.

Re: Why is my regex not matching for a multivalue field?

spike021 — Thu, 21 Jan 2016 15:53:43 GMT

Odd formatting.

So a typical event looks something like this. Priority is to get the keys from the "IMPORTANT" dictionary, but values as well in their own field would be very useful if I could get this to work properly

{
    "timestamp": "2016-01-21T14:44:28", 
    "SOME_FIELD": "etc.",
    "ANOTHER_FIELD": "...", 
    "IMPORTANT": {
        "a_string": 3,
        "another_strong": 44,
        "maybe_another...":95
    }, 
    "test": [
        [
            "something", 
            1.0
        ]
    ]
}

Re: Why is my regex not matching for a multivalue field?

spike021 — Thu, 21 Jan 2016 15:54:10 GMT

I just added a comment with a sample event.

Re: Why is my regex not matching for a multivalue field?

javiergn — Thu, 21 Jan 2016 15:59:37 GMT

Based on your code sample below why don't you try this (ignore the stats count and the eval as this is just for me to be able to generate an event):

| stats count
| eval myjson = "{
     \"timestamp\": \"2016-01-21T14:44:28\", 
     \"SOME_FIELD\": \"etc.\",
     \"ANOTHER_FIELD\": \"...\", 
     \"IMPORTANT\": {
         \"a_string\": 3,
         \"another_strong\": 44,
         \"maybe_another...\":95
     }, 
     \"test\": [
         [
             \"something\", 
             1.0
         ]
     ]
 }"
| spath input=myjson
| table IMPORTANT.*

Re: Why is my regex not matching for a multivalue field?

spike021 — Tue, 29 Sep 2020 08:31:32 GMT

Hmm. If I do that, it grabs each key of the dictionary, and turns them into the fields of the table. So I get a bunch of fields called "IMPORTANT_key_" and the integer value isn't shown at all.

Re: Why is my regex not matching for a multivalue field?

javiergn — Thu, 21 Jan 2016 16:11:31 GMT

What you do mean by "the integer value isn't shown at all".
If you copy and paste my code into your search box it should return the following:

IMPORTANT.a_string  IMPORTANT.another_strong    IMPORTANT.maybe_another...
3   44  95

Where you have field names and values in a table. You can then pipe it to "transpose" if you want to capture field names for instance.

Or maybe I'm completely misunderstanding your requirement.

Thanks,
J

Re: Why is my regex not matching for a multivalue field?

spike021 — Thu, 21 Jan 2016 16:19:25 GMT

Well I do get:

IMPORTANT.a_string     IMPORTANT.another_strong     IMPORTANT.maybe_another...

But the values aren't populating, plus these fields should then be separated per events rather than being actual field names. I think what's happening right now is it sets all the keys from IMPORTANT in the first event as the field names, but there are many events, each with different keys in IMPORTANT. So just setting it the current way doesn't work since those keys will usually be different.

Usually I'll probably do something like | table "timestamp", IMPORTANT

So that I get three columns. First would be timestamp since each event has a unique timestamp. Second column would be the keys, and third would be values. Or something similar to that.

Does that make more sense? Hopefully it does.

Re: Why is my regex not matching for a multivalue field?

spike021 — Thu, 21 Jan 2016 16:23:33 GMT

So I actually added an example event that you might have missed.

{
     "timestamp": "2016-01-21T14:44:28", 
     "SOME_FIELD": "etc.",
     "ANOTHER_FIELD": "...", 
     "IMPORTANT": {
         "a_string": 3,
         "another_strong": 44,
         "maybe_another...":95
     }, 
     "test": [
         [
             "something", 
             1.0
         ]
     ]
 }

Originally my idea just for the absolute minimum (to at least show I'm able to retrieve that part of the JSON data) was to use a | table "timestamp", IMPORTANT.key, IMPORTANT.values.

Maybe that isn't a good way to go about this?

Splunk already recognizes the rest of the fields at the top level. So if I do | table "timestamp", "ANOTHER_FIELD" then it works fine

Re: Why is my regex not matching for a multivalue field?

spike021 — Thu, 21 Jan 2016 16:26:20 GMT

My props/transforms are default right now since it seemed like Splunk could already pull out the top-level fields, as mentioned in my other comment a moment ago.

Maybe that's the problem.

Re: Why is my regex not matching for a multivalue field?

javiergn — Thu, 21 Jan 2016 16:28:47 GMT

Still don't get why values are empty but anyway, that's not the important thing.

Your field names are dynamic, so have you try to pipe your events to

 | spath input=yourjsonfieldname

Or:

 | spath input=IMPORTANT

Or maybe this in case all your event is in a Json format:

 | spath

Take a look at the command help. Spath is a brilliant command when you want to extract data from JSON or XML fields.