topic Re: SOURCE_KEY & multiple fields issue in Splunk Search

SOURCE_KEY & multiple fields issue

NaorPenso — Mon, 28 Sep 2020 16:48:46 GMT

Hi Everyone,
I have encountered an issue with SOURCE_KEY and MV_ADD
I need to extract multi-value fields (shown as FRAG's below)
the event looks like this:

*** 10 0 8 30 *NULL* foo 2 1 13671237459 11 1392550059 0 0 128928 4 3 3 0 bar *NULL* *NULL* 0 1 0 0 0 *NULL* 1 0 0 0 0 0 0 *NULL* 0 0 0 *NULL* 1607660 2 0 1440 0 0 1 1 0 1 0 *NULL* *NULL* 
FRAG 1 1 121238 0 0 0 0 1 2 457210 0 0 -1 0 3 0 0 6 1368450059 1234240476 0 *NULL* *NONE* 
FRAG 1 1 121258 0 0 0 0 1 2 187351 0 0 -1 0 3 0 0 6 1328450059 6235240476 0 *NULL* *NONE* 
FRAG 1 1 128518 0 0 0 0 1 2 262144 0 0 -1 0 3 0 0 6 1362410859 1233240476 0 *NULL* *NONE*

my Props.conf looks like this:

[foo] 
BREAK_ONLY_BEFORE = ***\s
MAX_TIMESTAMP_LOOKAHEAD = 150
NO_BINARY_CHECK = 1 pulldown_type = 1
REPORT-foo-a = foo-FRAG, foo-FRAG-fields

my transforms.conf looks like this:

[foo-FRAG]
SOURCE_KEY = _raw
REGEX      = (FRAG) ([^\r\n]+)
FORMAT     = $1::$2
MV_ADD     = true

[foo-FRAG-fields]
SOURCE_KEY = FRAG
DELIMS     = " "
FIELDS     = "field1","field2","field3","field4","field5"

Now the issue is, that the system recognizes the fields but does not treat them as multi-value fields.
Is there any resolution for this issue?

Thanks in advance!

Re: SOURCE_KEY & multiple fields issue

rsennett_splunk — Sat, 07 Jun 2014 17:12:54 GMT

There are a couple of things going on here.
First, the BREAK_ONLY_BEFORE directive receives a regex. So if you tried your ***\s in some thing like regexr.com it would tell you "nothing to repeat" what you want there is:

BREAK_ONLY_BEFORE= (*){3}\s

If you're testing with the data above... it just writes a complaint to the log and dumps the whole thing out as a multiline event... so you might not notice

You say above: "I need to extract multi-value fields (shown as FRAG's below)"

And your first transforms does just that:
[foo-FRAG] REGEX = (FRAG) ([^\r\n]+) FORMAT = $1::$2 MV_ADD = true
MV_ADD in this case is actually going to make a multivalue field out of FRAG

"If set to true, the extractor makes the field a multivalued field and appends the
newly found value, otherwise the newly found value is discarded."

So FRAG is a multivalue field...

you can see that if you do this in the search bar
eval n=mvcount(FRAG) | table FRAG n

But I suspect what you might want is what's INSIDE FRAG to be the multivalue field So that would happen only if FRAG was the event (SHOULD_LINEMERGE=false)
What you're probably seeing now, is that FRAG is extracted as a multivalue field and the first five values, of the first instance of the FRAG field, separated by a space, are deposited into field's called field1, field2, field3, field4 & field5

First... I see more than 5 fields... and it's unclear what you actually want in the fields other than the first five values of FRAG.

This answer... isn't really an answer, but the comments wouldn't hold all this.

you could use multikv.conf to be really specific about how you want to break this up but probably it would be easier to break things up within your search

Give us a visual example of what you want to end up with (be less foo/bar about it) and we'll edit the answer so others can benefit.

Re: SOURCE_KEY & multiple fields issue

NaorPenso — Sat, 07 Jun 2014 17:33:08 GMT

Hi rsennett,
Thanks a lot!
allow me to elaborate a bit without exposing sensitive data.

I am trying to analyze backup logs, each log contains the main event (aka ***) and within the event, fragments of the backup, where were they written to, how much was written etc.

as you suggested, FRAG doesn't matter, what's within is important. there are more than 5 fields, but as i just put 5 fields instead of 27 which is the correct amount of fields.

here is a picture of the results for "eval n=mvcount(FRAG) | table FRAG n"

Re: SOURCE_KEY & multiple fields issue

rsennett_splunk — Sun, 08 Jun 2014 17:32:09 GMT

One critical question and then I may have a suggestion for you. The "header" (the line prefixed with ***) has a timestamp (in epoch time). The FRAG lines also have what looks like a timestamp.

Which one will you use?

Also... we started this discussion about multivalue fields. However, in the end... it looks like what you really want to do, is break up the FRAG lines into individual fields. Yes?

Re: SOURCE_KEY & multiple fields issue

NaorPenso — Wed, 11 Jun 2014 10:31:01 GMT

Hi rsennett,
The header has several EPOC timestamps (start, end) further down the event.
The FRAG lines also has a time stamp, which may differ from the original time stamp, but the EPOC of the FRAG is less important to me.

I will be using the EPOC time of the header for this instance.

regarding the multivalue fields; I know that i can split the FRAG into different events but that will not be good as they must remain a part of the main event. mutlivalue is crucial to me as some of those values require mathematical manipulations (sum, avg etc.)

Thanks again for all of the assistance!!

Re: SOURCE_KEY & multiple fields issue

NaorPenso — Mon, 23 Jun 2014 15:48:53 GMT

Hi rsennett,
Any insights? I'm still stuck on this subject 😞

Thanks in advance,
Naor