Merge Events

[ckarthikin]

Hi,

Some of my events doesn't have an timestamp and its has been written as multiple line items in the log.

I want to merge the multiple line items into previous line item

Below are the examples, i want to merge the line items which doesn't have a timestamp, need to add it with previous line 

2024-05-24 14:11:51.7212|INFO|Services.Voice.VoiceManager|Wake word detected. hey_mentor
2024-05-24 14:11:51.7212|INFO|Services.Sound.SoundManager|Playing Sound.VoiceStart_TEMP
2024-05-24 14:11:53.9271|INFO|Services.Voice.VoiceManager|Received command
Spoken text: hey mentor turn off L E D
Intent Name: ChangeImageTransformOnOff
Intent Value: turn { OnOff off } { ImageTransformsOnOff L E D }
Slot 1: OnOff=off
Slot 2: ImageTransformsOnOff=L E D
2024-05-24 14:11:53.9271|INFO|NotificationService|Notify: [Illumination set to Off]
2024-05-24 14:11:59.5010|INFO|Services.Voice.VoiceManager|Wake word detected. hey_mentor
2024-05-24 14:11:59.5010|INFO|Services.Sound.SoundManager|Playing Sound.VoiceStart_TEMP
2024-05-24 14:12:01.8935|INFO|Services.Voice.VoiceManager|Received command
Spoken text: hey mentor turn on L E D
Intent Name: ChangeImageTransformOnOff
Intent Value: turn { OnOff on } { ImageTransformsOnOff L E D }
Slot 1: OnOff=on
Slot 2: ImageTransformsOnOff=L E D
2024-05-24 14:12:01.8935|INFO|NotificationService|Notify: [Illumination set to On]
2024-05-24 14:12:01.8935|INFO|Services.Sound.SoundManager|Playing Sound.VoiceStop_TEMP
2024-05-24 14:12:06.7081|INFO|Controls.Live.LiveModel|IsReady=True, Pause <<

 -------

Could any please help me how to write a query to achieve this.

[richgalloway]

This can be achieved using props.conf.  Try these settings to start with

[mysourcetype]
```The "Great Eight" settings```
SHOULD_LINEMERGE = false
```Break lines only between a line ending and a date (year)```
LINE_BREAKER = ([\r\n]+)\d{4}
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S.%4N
MAX_TIMESTAMP_LOOKAHEAD = 30
TRUNCATE = 10000
```Two settings for UFs```
EVENT_BREAKER_ENABLE = true
EVENT_BREAKER = ([\r\n]+)\d{4}

[ITWhisperer]

The events will have a value for _time. If you sort by this, are these events in the order you showed?

Your events which do have timestamps in the event seem to have the same timestamp for the events either side of the events without timestamps. Is this always the case?

For the events without timestamps, is each line in a different event of multiple events?

Please provide more detail so we can see what needs to be done and work out a solution for you.

[ckarthikin]

Hi, Thanks for the response.

we don't have _time but we have Time column (Indexed time - it will be same for all events so we cant use Time column). My expectation is without timestamp events need to be merged with previous events using any logic  need not save results and it will be used for some calculation and then it will be saved in saved search)

Yes always this is the case for all logs, so i need to write a query to transform this, Please help on this  and share your comments

[ITWhisperer]

Assuming your events follow the pattern shown, you could try something like this

| rex "[^\|]+\|(?<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2}\.\d{4})\|"
| streamstats count(time) as eventnumber
| stats values(time) as time list(_raw) as event by eventnumber
| eval _time=strptime(time,"%F %T.%4N")

This will also reset the _time timestamp to the same as found in the event data

[ckarthikin]

Hi Giuseppe,

Thank you for the response. Its coming from one of our device and these logs has been already uploaded to splunk and indexed already . now i want to write SPL query to merge the line items 

[gcusello]

Hi @ckarthikin ,

sorry but the issue is at ingestion level: you have to assign a correctly defined sourcetype (standard or custom) to your data, then you can search your data correctly parsed and aggregated.

so the questions are the ones before:

which technology?

which add-on used for parsing?

if none, you have to create a correct sourcetype and apply it to your data source.

Ciao.

Giuseppe

[gcusello]

Hi @ckarthikin ,

where do your logs come from? which technology?

did you used a standard add-on or not?

it seems to be a parsing error.

youshould try adding to your sourcetype SHOULD_LINEMERGE = True, in this way you configure a multiline sourcetype.

Ciao.

Giuseppe

[ckarthikin]

Hi Giuseppe,

Thank you for the response. Its coming from one of our device and these logs has been already uploaded to splunk and indexed already . now i want to write SPL query to merge the line items