topic Re: Filtering data using SHOULD_LINEMERGE in Getting Data In

Filtering data using SHOULD_LINEMERGE

danillopavan — Tue, 29 Sep 2020 17:12:02 GMT

Hi all,

I have configured the line breaking parameter as (SHOULD_LINEMERGE = true) to read a log file that contains the below data for each update. It will allow to have all data in the same event to run my searches:

S Doing: print 1597931/1
S --> Print Job @>SPOREQ:1597931@ S print job @>SPOREQ:1597931@ S replace user SAPSYS by NFEX
S print job @>SPOREQ:1597931@ S --------> db_rtab Error 128, table TSPEVJOB, action rspogdio_insert (expected)
S SpTSP01Select() - 1597931 is OTF/SMART/RDI/ADSx job, do not read TSP02L!
S Sß: (2017120211393600) sending job @>SPOREQ:1597931@DEV:JCD1@<'
S 1 processed

And I would like to filter the input data to index only the below lines of the log file disregarding the others lines:
S Doing: print 1597931/1
S replace user SAPSYS by NFEX

S Sß: (2017120211393600) sending job @>SPOREQ:1597931@DEV:JCD1@<'
S 1 processed

I was thinking to configure my REGEX just using the phases "Doing: print" , "replace user", "sending job", however it will not work for the line merge configuration as dont have the end of the line. So my question is: how I should configure the regex in transform file to allow just index 3 lines using SHOULD_LINEMERGE ?

I am using this below configuration on the props and transform files and it is not working:

[sourcetype]
SHOULD_LINEMERGE = true
BREAK_ONLY_BEFORE=processed
TRANSFORMS-set= setIndexnull,setIndexparsing

[setIndexnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setIndexparsing]
REGEX = print
DEST_KEY = queue
FORMAT = indexQueue

Thanks and regards,
Danillo Pavan

Re: Filtering data using SHOULD_LINEMERGE

danillopavan — Sat, 09 Dec 2017 18:36:56 GMT

I am using this below configuration on the transform files and it is not working:

[setIndexnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[setIndexparsing]
REGEX = print
DEST_KEY = queue

FORMAT = indexQueue

Re: Filtering data using SHOULD_LINEMERGE

sandyIscream — Tue, 12 Dec 2017 13:45:36 GMT

Write your props and transforms.conf like the below and try this.

props.conf

[sourcetype]
TRANSFORMS-set = setnull, print1,print2,print3

Transforms.conf

[setnull]
REGEX = .
DEST_KEY = queue
FORMAT = nullQueue

[print1]
REGEX = S\sDoing:
DEST_KEY = queue
FORMAT = indexQueue

[print2]
REGEX = S\sreplace
DEST_KEY = queue
FORMAT = indexQueue

[print3]
REGEX = S \sSß:
DEST_KEY = queue
FORMAT = indexQueue

Re: Filtering data using SHOULD_LINEMERGE

danillopavan — Tue, 12 Dec 2017 14:20:39 GMT

Hello Sandy,

Now no lines are being indexed. The regex commands that you suggested seems to be wrong. Anyway, the instructions that you suggested, in my opinion, would not work. I need to conciliate MERGED LINES with FILTERING.

Re: Filtering data using SHOULD_LINEMERGE

danillopavan — Sat, 16 Dec 2017 15:04:37 GMT

Any answer?

Re: Filtering data using SHOULD_LINEMERGE

niketn — Sat, 16 Dec 2017 18:47:46 GMT

@danillopavan this seems similar to other question your have posted: https://answers.splunk.com/answers/597389/filtering-data-using-should-linemerge.html

I would request you to consolidate required details against single question and keep only one of them open.

Re: Filtering data using SHOULD_LINEMERGE

danillopavan — Tue, 19 Dec 2017 21:02:07 GMT

I still didn´t find any solution for this case. Still need to trash certain lines and then create multiline events out of what is left...
Any proposal??