How can I merge events based on Start and End value pairs when those values differ each time?

[Pradeepkandukoo]

How can I merge events based on Start and End value pairs when Start and end value differs each time?

For example: I have events as below.
Timestamp5 End Ticket: 2
Timestamp4 data3
Timestamp3 data2
Timestamp2 data1
Timestamp1 Start Ticket: 2
Timestamp5 End Ticket: 1
Timestamp4 data3
Timestamp3 data2
Timestamp2 data1
Timestamp1 Start Ticket: 1

So now i want group these events into one event based on the Ticket number in real time.

Can some one suggest how we can do it?

[solarboyz1]

If there is a ticketID field in the events already, the easiest way is to use that field to join the events.

| transaction ticketID startswith="Start Ticket" endswith="End Ticket" 

However, there are limits to the transaction function.
You can also use stats, to similarly join:

| stats values(data1), values(data2), sum(data3), min(_time) as Start, max(_time) as Stop by ticketID

stats won't have the same limits as transaction.

[woodcock]

Like this:

| makeresults | eval raw="Timestamp5 End Ticket: 2:::Timestamp4 data3:::Timestamp3 data2:::Timestamp2 data1:::Timestamp1 Start Ticket: 2:::Timestamp5 End Ticket: 1:::Timestamp4 data3:::Timestamp3 data2:::Timestamp2 data1:::Timestamp1 Start Ticket: 1"
| makemv delim=":::" raw
| mvexpand raw
| rename raw AS _raw
| fields - _time

| rename COMMENT AS "Everything above generates sample event data; everything below is your solution"

| streamstats count(eval(searchmatch("End Ticket"))) AS SessionID
| reverse
| list(_raw) AS events BY SessionID

[shin_matsuzawa]

Hi,

I was surprised that there was such a way.
I'm interested in it.

I didn't know the "streamstats" command and I will try this command in my environment.

I appreciate your idea.
Thank you.

[woodcock]

Don't forget to come back here to UpVote and click Accept if you get a good answer.

[richgalloway]

Ignoring the 'real time' part of the question, it seems a transaction will help. Try this

index = foo | transaction startswith="Start Ticket" endswith="End Ticket" | ...

Be aware that transaction is a slow command. Like @adonio said, if you can provide more information about your end goal, we may be able to offer a better answer.

[cpetterborg]

As Rich says, "ignoring real time". You don't want to do a transaction command in a real-time search, it will just mess things up. Hopefully you are meaning that you want to just get the results at search time, not index time when you say "real time."

[shin_matsuzawa]

Hi,

Do you mean that you want to put together the events with the same Ticket?
Do you have any element?

Following is an example:

2018-12-13 14:08:24,281 id-bbb End Ticket: 2
2018-12-13 14:07:24,281 id-bbb data3
2018-12-13 14:06:24,281 id-bbb data2
2018-12-13 14:05:24,281 id-bbb data1
2018-12-13 14:04:24,281 id-bbb Start Ticket: 2
2018-12-13 14:08:24,281 id-aaa End Ticket: 1
2018-12-13 14:07:24,281 id-aaa data3
2018-12-13 14:06:24,281 id-aaa data2
2018-12-13 14:05:24,281 id-aaa data1
2018-12-13 14:04:24,281 id-aaa Start Ticket: 1

If your events have elements like a "id-aaa", you can combine events using "transaction" command.

ex.

source="test.log" host="test01" sourcetype="test"
| rex field=_raw "(?ms)^[^,\\n]*,\\d+\\s+(?P<id>[^ ]+)"
| transaction host id startswith="Start" endswith="End"

Is my image correct?

[adonio]

please elaborate,
what is the problem you are trying to solve?
when you say "group these events to one event", what exactly do you mean?
what are you intend doing with that "group"?
do you need the events that have the start and end ticket as well?