topic Re: hot to merge multiple lines into a single event? in Getting Data In

hot to merge multiple lines into a single event?

0xlc — Wed, 30 Sep 2020 00:28:46 GMT

hi guys,

i am trying to merge these lines into a event

so far i tried

[cycledata]
EVENT_BREAKER = (CycleDataTask finished)
SHOULD_LINEMERGE = false

i got block of lines starting with CycleDataTask started and finishing with CycleDataTask finished and i want to group them into a single event for each started finished.

and MUST BREAK AFTER same regex

these is an example:

2019-05-09 13:29:02.3975 INFO CycleData - CycleDataTask started ________________________________________________________
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS:-> =
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS: -> 
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS: -> 
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - CycleDataTask finished _______________________________________________________

thank you

Re: hot to merge multiple lines into a single event?

richgalloway — Thu, 09 May 2019 19:57:47 GMT

Is the example a single event you want to break into multiple events or multiple events you want to make into a single event?

Re: hot to merge multiple lines into a single event?

PowerPacked — Wed, 30 Sep 2020 00:27:46 GMT

Just check if you gave (EVENT_BREAKER) instead of LINE_BREAKER attribute

[cycledata]
LINE_BREAKER = (CycleDataTask finished)
SHOULD_LINEMERGE = false

the above should work

Thanks

Re: hot to merge multiple lines into a single event?

0xlc — Fri, 10 May 2019 08:37:28 GMT

the example has been edited, and like that seems just one long line, instead i have multiple lines which i want to merge in a single event

it should start here:

2019-05-09 13:29:02.3975 INFO CycleData - CycleDataTask started ________________________________________________________

and finish here:

2019-05-09 13:29:06.8166 INFO CycleData - CycleDataTask finished _______________________________________________________

Re: hot to merge multiple lines into a single event?

0xlc — Fri, 10 May 2019 09:05:58 GMT

no it does not work

i almost find a solution adding the example in data file but now it cuts off the word 'finished' which i use as regex. and of course i need it in.

(finished)

Re: hot to merge multiple lines into a single event?

koshyk — Fri, 10 May 2019 10:59:40 GMT

For sample data

2019-05-09 14:41:02.3975 INFO CycleData - CycleDataTask started
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - CycleDataTask finished

Please find solution

[cycledata]
LINE_BREAKER = CycleDataTask finished([\r\n]+)
SHOULD_LINEMERGE = false

cheers

Re: hot to merge multiple lines into a single event?

0xlc — Fri, 10 May 2019 13:11:47 GMT

this does not work

Re: hot to merge multiple lines into a single event?

0xlc — Fri, 10 May 2019 13:12:53 GMT

i think there is something going on with my cluster, if i upload a txt sample, and i add the regex (finished) it merges it almost fine, but then, when i add to props.conf does not work at all.

Re: hot to merge multiple lines into a single event?

koshyk — Fri, 10 May 2019 13:13:52 GMT

eh? Can you please verify if the sample data is like above?
Your example had all lines merged already, so no settings required. I have split that into individual lines

Please put your sample data again in a formatted way as it exactly occurs in your file (not in Splunk)

Re: hot to merge multiple lines into a single event?

0xlc — Fri, 10 May 2019 13:26:14 GMT

check now please

reading again my message i was not very clear, i edited it.

Re: hot to merge multiple lines into a single event?

sanjeev543 — Fri, 10 May 2019 13:45:48 GMT

I took below sample data and verified the config that I mentioned

2019-05-09 14:41:02.3975 INFO CycleData - CycleDataTask started
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - CycleDataTask finished
 2019-05-10 14:41:02.3975 INFO CycleData - CycleDataTask started
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - CycleDataTask finished
 2019-05-08 14:41:02.3975 INFO CycleData - CycleDataTask started
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - CycleDataTask finished

Config I have tried

LINE_BREAKER = CycleDataTask\sfinished([\r\n]*)
 MUST_BREAK_AFTER  = CycleDataTask\sfinished
 SHOULD_LINEMERGE = true
 TIME_PREFIX = ^
 MAX_TIMESTAMP_LOOKAHEAD = 26
 TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4N

This is parsing as you are expecting, hopefully this helps

Re: hot to merge multiple lines into a single event?

0xlc — Fri, 10 May 2019 13:55:34 GMT

it's not working for me, as i said below, i believe there is something wrong going on with my cluster and config files.

as soon i find a solution and this work i'll mark it as working

thank you

this is how it looks like

link text