Ordering events in transaction

[datasearchninja]

I have some logs arriving via syslog, that have a single event broken up into multiple syslog messages. Due to issues with time and timezones at the source, the syslog server is stamping the messages with a new timestamp. Due to the distributed nature of this syslog setup, it is technically possible for the events to arrive out of order.

Events look like this: (In this example the events did arrive in order)

2013-03-14T23:14:20.497362+00:00 DEVICENAME CategoryName MessageCode ID 4 Message Text
2013-03-14T23:14:20.497355+00:00 DEVICENAME CategoryName MessageCode ID 3 Message Text
2013-03-14T23:14:20.497355+00:00 DEVICENAME CategoryName MessageCode ID 2 Message Text
2013-03-14T23:14:20.497223+00:00 DEVICENAME CategoryName MessageCode ID 1 Message Text
2013-03-14T23:14:20.497223+00:00 DEVICENAME CategoryName MessageCode ID 0 Message Text

Is it possible to order events in a transaction, based on the event line number? (Above these are 0-4)

This search forces _time to be modified by the value of the line number, which then ensures they are ordered correctly by time. Is there a better way?

| eval EventLineCounter=0 | rex "^[^ ]+ [^ ]+ [^ ]+ (?<MessageCode>[0-9]+) [0-9]+ (?<EventLineCounter>[0-9]+)" | eval _time=_time+tonumber(EventLineCounter) | sort 0 _time | transaction host MessageCode maxspan=30s

[kristian_kolb]

I don't think you need the sort, since that is done through the transaction.
Also, you'd need to set the new _time properly. In essence you're right, but I'd do it slightly different; stripping the sub-second part of the timestamp (i.e splitting on the dot) and replacing it with the ID-number (i.e. appending a dot followed by the EventLineCounter). Remember that the dot is also the string concatenation operator, which makes that line look a bit 'funny'.

your search 
| eval new_t = split(_time,".") 
| eval new_t=mvindex(new_t,0).".".EventLineCounter 
| the rest of your search

Hope this helps,

Kristian

[datasearchninja]

Actually, I think transaction assumes that the events coming in are sorted. Certainly I get different results with and without the sort.

I also don't understand how 'new_t' is used in the transaction. Did you mean to eval _time in the second eval in your example?