How to extract fields from a single file containing multiple sourcetypes, each with multiline events ?

[mzorzi]

My source file is like:

============================

App01trace

3 0 393222 0 19 148 8838300

4 0 458759 0 29 15 0

4 0 458759 0 31 12 0

5 0 524296 0 61 170 8869500

App02trace

4 0 327685 2032 0 0 0 0 NULL

6 0 393222 2032 0 0 0 0 NULL

5 0 458760 2032 0 0 0 0 NULL

App03trace

21 1 2959165 3 8 1 1 P

22 9 859165 3 12 6 1 R

============================

I would like to associate to each App0.trace a different sourcetype, and then associate each value to a different field, specific for that sourcetype.

I've tried the following steps:

1. In inputs.conf I assign my source file to a fix sourcetype [testbb]

2. I then use [testbb] to define a stanza props.conf, like this:

   [testbb]
   
   SHOULD_LINEMERGE = true
   
   BREAK_ONLY_BEFORE_DATE = false
   
   LINE_BREAKER = ([\r\n]+)(App01trace|App02trace|App03trace)
   
   TRANSFORMS-App01trace=tr-App01trace
   
   TRANSFORMS-App02trace=tr-App02trace
   
   TRANSFORMS-App03trace=tr-App03trace

3. In transforms.conf I extract the three different sourcetypes, like:

   [tr-App01trace]
   
   REGEX = App01trace
   
   DEST_KEY = MetaData:Sourcetype
   
   FORMAT = sourcetype::App01trace

4. Finally I create another stanza in props.conf for sourcetype [App01trace] where I perform the search time extractions (using Extract)

My problem is that it only extracts the fields from the first line of each sub block, not for every line. So supposing the third field for App03trace is called app03field3, I get app03field3=2959165 but the value 859165 doesn't get extracted.

That doesn't surprise me as the LINE_BREAKER has been already executed for that stream.

What should I change to achieve my goal?

Thanks for your help.

[mzorzi]

Hi David,

thanks for your answer, I've tried and it works great!

What I forgot to mention is that I would need to have an event for each line; so if I search for sourcetype=App01trace and field3=393222 and field5=19, I should get only this line

3 0 393222 0 19 148 8838300

[hazekamp]

mzorzi,

You may want to change to using PROPS instead of EXTRACT. This way you can specify MV_ADD=True which will create multivalue fields based on your search time extraction.

Given a sample of:

App01trace
3 0 393222 0 19 148 8838300
4 0 458759 0 29 15 0
4 0 458759 0 31 12 0
5 0 524296 0 61 170 8869500 

The following configuration:

## props.conf
[App01trace]
REPORT-kv_for_app01trace = kv_for_app01trace

## transforms.conf
[kv_for_app01trace]
REGEX = ([^\s]+)\s([^\s]+)\s([^\s]+)\s([^\s]+)\s([^\s]+)\s([^\s]+)\s([^\s]+)
FORMAT = field1::$1 field2::$2 field3::$3 field4::$4 field5::$5 field6::$6 field7::$7
MV_ADD = True

Will give you the following fields and values:

field1 = [3, 4, 4, 5]
field2 = [0, 0, 0, 0]
field3 = [393222, 458759, 458759, 524296]
filed4 = [0, 0, 0, 0]
field5 = [19, 29, 31, 61]
field6 = [148, 15, 12, 170]
field7 = [8838300, 0, 0, 8869500]

There are additional ways of performing extractions using search commands, but this is the best way to do via props.conf.

-David