Getting Data In
×

Are you a member of the Splunk Community?

Sign in or Register with your Splunk account to get your questions answered, access valuable resources and connect with experts!
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Ask a Question

hot to merge multiple lines into a single event?

0xlc
Path Finder
‎05-09-2019 09:06 AM

hi guys,

i am trying to merge these lines into a event

so far i tried

[cycledata]
EVENT_BREAKER = (CycleDataTask finished)
SHOULD_LINEMERGE = false

i got block of lines starting with CycleDataTask started and finishing with CycleDataTask finished and i want to group them into a single event for each started finished.

and MUST BREAK AFTER same regex

these is an example:

2019-05-09 13:29:02.3975 INFO CycleData - CycleDataTask started ________________________________________________________
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS:-> =
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS: -> 
2019-05-09 13:29:06.3746 INFO CycleData - Pool has NEW TICKETS: -> 
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 13:29:06.8166 INFO CycleData - CycleDataTask finished _______________________________________________________

thank you

0 Karma
Reply

sanjeev543
Communicator
‎05-10-2019 06:45 AM

I took below sample data and verified the config that I mentioned 

2019-05-09 14:41:02.3975 INFO CycleData - CycleDataTask started
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-09 14:41:06.8166 INFO CycleData - CycleDataTask finished
 2019-05-10 14:41:02.3975 INFO CycleData - CycleDataTask started
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-10 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-10 14:41:06.8166 INFO CycleData - CycleDataTask finished
 2019-05-08 14:41:02.3975 INFO CycleData - CycleDataTask started
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-08 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
 2019-05-08 14:41:06.8166 INFO CycleData - CycleDataTask finished

Config I have tried

LINE_BREAKER = CycleDataTask\sfinished([\r\n]*)
 MUST_BREAK_AFTER  = CycleDataTask\sfinished
 SHOULD_LINEMERGE = true
 TIME_PREFIX = ^
 MAX_TIMESTAMP_LOOKAHEAD = 26
 TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4N

This is parsing as you are expecting, hopefully this helps

0 Karma
Reply

0xlc
Path Finder
‎05-10-2019 06:55 AM

it's not working for me, as i said below, i believe there is something wrong going on with my cluster and config files.

as soon i find a solution and this work i'll mark it as working

thank you

this is how it looks like

link text

0 Karma
Reply

0xlc
Path Finder
‎05-10-2019 06:12 AM

i think there is something going on with my cluster, if i upload a txt sample, and i add the regex (finished) it merges it almost fine, but then, when i add to props.conf does not work at all.

0 Karma
Reply

koshyk
Super Champion
‎05-10-2019 03:59 AM

For sample data

2019-05-09 14:41:02.3975 INFO CycleData - CycleDataTask started
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS:-> 
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 14:41:06.3746 INFO CycleData - Pool has NEW TICKETS: ->
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - Pool has been updated succesfully.
2019-05-09 14:41:06.8166 INFO CycleData - CycleDataTask finished

Please find solution

[cycledata]
LINE_BREAKER = CycleDataTask finished([\r\n]+)
SHOULD_LINEMERGE = false

cheers

0 Karma
Reply

0xlc
Path Finder
‎05-10-2019 06:11 AM

this does not work

0 Karma
Reply

koshyk
Super Champion
‎05-10-2019 06:13 AM

eh? Can you please verify if the sample data is like above?
Your example had all lines merged already, so no settings required. I have split that into individual lines

Please put your sample data again in a formatted way as it exactly occurs in your file (not in Splunk)

0 Karma
Reply

0xlc
Path Finder
‎05-10-2019 06:26 AM

check now please

reading again my message i was not very clear, i edited it.

0 Karma
Reply

PowerPacked
Builder
‎05-09-2019 04:16 PM

Hi

Just check if you gave (EVENT_BREAKER) instead of LINE_BREAKER attribute

[cycledata]
LINE_BREAKER = (CycleDataTask finished)
SHOULD_LINEMERGE = false

the above should work

Thanks

0 Karma
Reply

0xlc
Path Finder
‎05-10-2019 02:05 AM

no it does not work

i almost find a solution adding the example in data file but now it cuts off the word 'finished' which i use as regex. and of course i need it in.

(finished)

0 Karma
Reply

richgalloway
SplunkTrust
SplunkTrust
‎05-09-2019 12:57 PM

Is the example a single event you want to break into multiple events or multiple events you want to make into a single event?

---
If this reply helps you, Karma would be appreciated.
0 Karma
Reply

0xlc
Path Finder
‎05-10-2019 01:37 AM

the example has been edited, and like that seems just one long line, instead i have multiple lines which i want to merge in a single event

it should start here:

2019-05-09 13:29:02.3975 INFO CycleData - CycleDataTask started ________________________________________________________

and finish here:

2019-05-09 13:29:06.8166 INFO CycleData - CycleDataTask finished _______________________________________________________

0 Karma
Reply
Career Survey
First 500 qualified respondents will receive a $20 gift card! Tell us about your professional Splunk journey.
Take the Survey
Get Updates on the Splunk Community!

Tech Talk Recap | Mastering Threat Hunting

Mastering Threat HuntingDive into the world of threat hunting, exploring the key differences between ...

Observability for AI Applications: Troubleshooting Latency

If you’re working with proprietary company data, you’re probably going to have a locally hosted LLM or many ...

Splunk AI Assistant for SPL vs. ChatGPT: Which One is Better?

In the age of AI, every tool promises to make our lives easier. From summarizing content to writing code, ...